You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Franklin Hu (JIRA)" <ji...@apache.org> on 2011/08/09 03:00:37 UTC
[jira] [Created] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
create dynamic partition if and only if intermediate source has files
---------------------------------------------------------------------
Key: HIVE-2360
URL: https://issues.apache.org/jira/browse/HIVE-2360
Project: Hive
Issue Type: Bug
Reporter: Franklin Hu
Assignee: Franklin Hu
Priority: Minor
Fix For: 0.8.0
There are some race conditions in which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Franklin Hu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Franklin Hu updated HIVE-2360:
------------------------------
Attachment: hive-2360.2.patch
refactor to use one call to Utilities.getFileStatusRecurse()
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong reassigned HIVE-2360:
---------------------------------
Assignee: (was: Franklin Hu)
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2360:
---------------------------------
Fix Version/s: (was: 0.8.0)
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Priority: Minor
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Franklin Hu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Franklin Hu updated HIVE-2360:
------------------------------
Attachment: hive-2360.1.patch
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081780#comment-13081780 ]
Ning Zhang commented on HIVE-2360:
----------------------------------
Franklin, can you create a review request? That's much easier to comment. Two comments here:
1) in dynamic partition inserts, a subquery corresponding to a partition is created only if there is a row that should be inserted into that partition, which means the directory should not be empty. If the directory is empty, most likely there is a bug there and we should fix that bug instead.
2) the function Utilities.getFileStatusRecurse() is quite expensive. We should avoid using it and use cached results (it's called somewhere else before).
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103145#comment-13103145 ]
Siying Dong commented on HIVE-2360:
-----------------------------------
Franklin finished his internship and left. We should find another one to finish the task.
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081995#comment-13081995 ]
jiraposter@reviews.apache.org commented on HIVE-2360:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1442/
-----------------------------------------------------------
Review request for hive, Ning Zhang and Siying Dong.
Summary
-------
There are some conditions under which partition descriptions are created in memory and committed to the metastore despite there being no intermediate or final finals in that directory (due to dynamic partitioning).
In this change, a check is done to only call loadPartitions that have files in them.
This addresses bug HIVE-2360.
https://issues.apache.org/jira/browse/HIVE-2360
Diffs
-----
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1155968
Diff: https://reviews.apache.org/r/1442/diff
Testing
-------
Unit tests pass
Thanks,
Franklin
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Kaushik Lakshminarayanan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132289#comment-13132289 ]
Kaushik Lakshminarayanan commented on HIVE-2360:
------------------------------------------------
The patch (https://issues.apache.org/jira/secure/attachment/12489913/hive-2360.2.patch) passes all the unit tests.
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Priority: Minor
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135732#comment-13135732 ]
Hudson commented on HIVE-2360:
------------------------------
Integrated in Hive-trunk-h0.21 #1035 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1035/])
HIVE-2360 create dynamic partition if and only if intermediate source has files (Franklin Hu, Kaushik Lakshminarayanan via Siying Dong)
sdong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188989
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Priority: Minor
> Attachments: hive-2360.1.patch, hive-2360.2.patch
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2360) create dynamic partition if and only
if intermediate source has files
Posted by "Franklin Hu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Franklin Hu updated HIVE-2360:
------------------------------
Description: There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files). (was: There are some race conditions in which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).)
> create dynamic partition if and only if intermediate source has files
> ---------------------------------------------------------------------
>
> Key: HIVE-2360
> URL: https://issues.apache.org/jira/browse/HIVE-2360
> Project: Hive
> Issue Type: Bug
> Reporter: Franklin Hu
> Assignee: Franklin Hu
> Priority: Minor
> Fix For: 0.8.0
>
>
> There are some conditions under which a partition description is created due to insert overwriting a table using dynamic partitioning for partitions that that are empty (have no files).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira