You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2010/06/19 02:10:23 UTC
[jira] Created: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
------------------------------------------------------------------------------
Key: HIVE-1416
URL: https://issues.apache.org/jira/browse/HIVE-1416
Project: Hadoop Hive
Issue Type: Bug
Reporter: Ning Zhang
Assignee: Ning Zhang
Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1416:
-----------------------------
Attachment: HIVE-1416.2.patch
new patch that removes accidental junks in HadoopShims.java.
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1416:
-----------------------------
Status: Open (was: Patch Available)
Ning, this ran through cleanly with Hadoop 0.17 (where I verified that it fixes the problem), but on Hadoop 0.20, it results in a lot of test failures. These aren't just diffs due to missing ORDER BY; values are actually missing from the results.
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881440#action_12881440 ]
HBase Review Board commented on HIVE-1416:
------------------------------------------
Message from: "Ning Zhang" <n....@gmail.com>
bq. On 2010-06-22 15:34:57, John Sichi wrote:
bq. > http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 408
bq. > <http://review.hbase.org/r/223/diff/1/?file=1551#file1551line408>
bq. >
bq. > Rather than repeating the HiveConf.getVar in several places, it would be cleaner to just pass the configuration down into the Utilities method as the new parameter and have it do the configuration check.
I didn't that before, but changed the way in the patch later. The reason is that the getting the value of localMode is HiveConf.getVar is a hash lookup and and a string comparison. It is quite expensive if it is called many times. In the current patch, the HiveConf.getVar() and string comparison are called only once and passed to the for-loop.
- Ning
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/223/#review268
-----------------------------------------------------------
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1416:
-----------------------------
Attachment: junit-noframes.html
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch, junit-noframes.html
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881419#action_12881419 ]
HBase Review Board commented on HIVE-1416:
------------------------------------------
Message from: "John Sichi" <js...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/223/
-----------------------------------------------------------
Review request for Hive Developers.
Summary
-------
jvs reviewing Ning's change
This addresses bug HIVE-1416.
http://issues.apache.org/jira/browse/HIVE-1416
Diffs
-----
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/0.17/java/org/apache/hadoop/hive/shims/Hadoop17Shims.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/0.18/java/org/apache/hadoop/hive/shims/Hadoop18Shims.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/0.19/java/org/apache/hadoop/hive/shims/Hadoop19Shims.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 956664
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 956664
Diff: http://review.hbase.org/r/223/diff
Testing
-------
Thanks,
John
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881452#action_12881452 ]
John Sichi commented on HIVE-1416:
----------------------------------
I think a profiler would show it as negligible in that context, but I won't argue the point. Can you fix the other one?
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881468#action_12881468 ]
John Sichi commented on HIVE-1416:
----------------------------------
+1. Will commit when tests pass.
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881921#action_12881921 ]
John Sichi commented on HIVE-1416:
----------------------------------
Attached junit-noframes.html with the failures (but not the diffs).
Example diff snippet from union6.q:
@@ -233,7 +233,6 @@
406 val_406
66 val_66
98 val_98
-tst1 500
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch, junit-noframes.html
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881426#action_12881426 ]
HBase Review Board commented on HIVE-1416:
------------------------------------------
Message from: "John Sichi" <js...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/223/#review268
-----------------------------------------------------------
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
<http://review.hbase.org/r/223/#comment1126>
Rather than repeating the HiveConf.getVar in several places, it would be cleaner to just pass the configuration down into the Utilities method as the new parameter and have it do the configuration check.
http://svn.apache.org/repos/asf/hadoop/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
<http://review.hbase.org/r/223/#comment1124>
This "anObject" insertion looks like it was accidental.
- John
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881439#action_12881439 ]
HBase Review Board commented on HIVE-1416:
------------------------------------------
Message from: "Ning Zhang" <n....@gmail.com>
bq. On 2010-06-22 15:34:57, John Sichi wrote:
bq. > http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 408
bq. > <http://review.hbase.org/r/223/diff/1/?file=1551#file1551line408>
bq. >
bq. > Rather than repeating the HiveConf.getVar in several places, it would be cleaner to just pass the configuration down into the Utilities method as the new parameter and have it do the configuration check.
bq.
bq. Ning Zhang wrote:
bq. I didn't that before, but changed the way in the patch later. The reason is that the getting the value of localMode is HiveConf.getVar is a hash lookup and and a string comparison. It is quite expensive if it is called many times. In the current patch, the HiveConf.getVar() and string comparison are called only once and passed to the for-loop.
I meant "I did that before...".
- Ning
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/223/#review268
-----------------------------------------------------------
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang resolved HIVE-1416.
------------------------------
Resolution: Not A Problem
This issue is fixed by another patch that fixes the code to extract task ID from file names for hadoop 0.17. It is not an issue in both branch 0.6 and trunk (which dropped support for pre-0.20 hadoop).
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch, junit-noframes.html
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1416:
-----------------------------
Fix Version/s: (was: 0.6.0)
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1416.2.patch, HIVE-1416.patch, junit-noframes.html
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1416:
-----------------------------
Attachment: HIVE-1416.patch
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1416) Dynamic partition inserts left empty
files uncleaned in hadoop 0.17 local mode
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1416:
-----------------------------
Status: Patch Available (was: Open)
Affects Version/s: 0.6.0
0.7.0
Fix Version/s: 0.6.0
0.7.0
> Dynamic partition inserts left empty files uncleaned in hadoop 0.17 local mode
> ------------------------------------------------------------------------------
>
> Key: HIVE-1416
> URL: https://issues.apache.org/jira/browse/HIVE-1416
> Project: Hadoop Hive
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1416.patch
>
>
> Hive parses the file name generated by tasks to figure out the task ID in order to generate files for empty buckets. Different hadoop versions and execution mode have different ways of naming output files by mappers/reducers. We need to move the parsing code to shims.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.