You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2009/06/05 01:01:07 UTC
[jira] Created: (HIVE-543) provide option to run hive in local mode
provide option to run hive in local mode
----------------------------------------
Key: HIVE-543
URL: https://issues.apache.org/jira/browse/HIVE-543
Project: Hadoop Hive
Issue Type: Bug
Reporter: Joydeep Sen Sarma
this is a little bit more than just mapred.job.tracker=local
when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
hadoop.tmp.dir
mapred.local.dir
need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-543) provide option to run hive in local
mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879039#action_12879039 ]
Joydeep Sen Sarma commented on HIVE-543:
----------------------------------------
#2 - this piece of code got quite messed up because of the changes because of parallel execution (hive-549). the initial synchronized block protected access to a single global variable. this was replaced by a synchronized map (gWorkMap) - but the surrounding synchronized block was never taken out (it's now unnecessary because of the synchronized map). also the (gWork==null) check inside the synchronized section was redundant (it made sense when there was a singleton pattern - but with the synchronized map doesn't make sense).
#1- don't understand this. there's already a hadoop parameter (mapred.local.dir) for specifying local scratch directory. i didn't add any new parameters as far as hive scratch directories are concerned .. is the concern about automatically selecting local intermediate directory for local mode execution? - that should be ok.
#3 - mystery to me as well. the ordering of the output lines has changed (not the content). the diff script is not able to ignore these changes.
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-534.patch.2, hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-543) provide option to run hive in local
mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma reassigned HIVE-543:
--------------------------------------
Assignee: Joydeep Sen Sarma (was: He Yongqiang)
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-543) provide option to run hive in local
mode
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reassigned HIVE-543:
-------------------------------
Assignee: He Yongqiang
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: He Yongqiang
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-543:
-----------------------------------
Attachment: hive-534.patch.2
log jobconf parameters from child jvm in local mode. emulates the fact that we can look at job.xml from TT/JT.
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-534.patch.2, hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-543:
--------------------------------
Affects Version/s: 0.5.0
Component/s: Query Processor
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.5.0
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Fix For: 0.6.0
>
> Attachments: hive-534.patch.2, hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-543:
-----------------------------------
Attachment: hive-543.patch.1
a few fixes for better local mode execution:
- provide alternate log4j file configuration for capturing local mode execution log (and default to hive log4j if none provided). this cleans up the goop on the cli but allows captuing execution time logs in a separate location if desired
- bypass distributed cache for local mode submissions. saves on hdfs time
- some cleanup on the set/get MapRedWork code path. it seems to have been messed up after the parallel execution changes
- getMRScratchDir - now returns a local scratch dir when executing in local mode. so we don't hit hdfs unnecessarily in local mode.
- fix to fileutils.makequalified because of the above. there was a subtle bug in this that was causing file paths to get messed up when using local paths for interemediate data
- bypassed query plan serialization/deserialization except for test mode. from past experience - xml serialization/deserialization is pretty expensive and makes no sense to subject every query to it.
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-543:
----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed. Thanks Joy
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.5.0
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Fix For: 0.6.0
>
> Attachments: hive-534.patch.2, hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-543:
-----------------------------------
Status: Patch Available (was: Open)
- mapred.job.tracker=local continues to be the way to setup hive local mode.
- admins can provide appropriate mapred.local.dir and mapred.system.dir settings for hive clients (for local mode execution). they can do this either via hive configuration files or via hadoop client-side-only configuration files. for regular cluster jobs - these are controlled by hadoop server side configuration files.
some of the cleanups regarding randomizing local/system directories etc. for concurrent queries were already in place (via hive-77).
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-543) provide option to run hive in local mode
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-543:
----------------------------
Issue Type: Improvement (was: Bug)
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-543) provide option to run hive in local
mode
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879002#action_12879002 ]
Namit Jain commented on HIVE-543:
---------------------------------
Some minor comments:
1. Do you want to add a parameter for specifying the local directory ?
2. Not sure about the removal of synchronization around gWorkMap - let us go over it offline.
3. Need to see in more detail - why did the output change for only 2 join tests
> provide option to run hive in local mode
> ----------------------------------------
>
> Key: HIVE-543
> URL: https://issues.apache.org/jira/browse/HIVE-543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: hive-534.patch.2, hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.