You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2010/08/05 10:01:16 UTC
[jira] Created: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
hive starter scripts should load admin/user supplied script for configurability
-------------------------------------------------------------------------------
Key: HIVE-1513
URL: https://issues.apache.org/jira/browse/HIVE-1513
Project: Hadoop Hive
Issue Type: Improvement
Components: CLI
Reporter: Joydeep Sen Sarma
it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896231#action_12896231 ]
Joydeep Sen Sarma commented on HIVE-1513:
-----------------------------------------
HADOOP_HEAPSIZE: hive-config.sh only supplies a default. if the admin has specified a value in hive-env - it will be used instead.
HADOOP_OPTS: seems like it's appending a specific JVM flag. i agree this doesn't make sense (the admin should choose whether they want that flag or not). i will post another patch after taking it out.
not sure about whether we should rename the template to .sh. hadoop-20 seems to have template files only.
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch, 1513.2.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1513:
------------------------------------
Attachment: hive-1513.3.patch
Ning - u are right - i didn't notice the HEAP setting. in the internal tree - the heap setting is done differently (and i thought that the internal tree does not override these scripts).
so i have incorporated both the suggestions (don't set opts/heap). also - the build script needed a slight change to make the .template file part of distribution.
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch, 1513.2.patch, hive-1513.3.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1513:
------------------------------------
Status: Patch Available (was: Open)
Assignee: Joydeep Sen Sarma
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1513:
------------------------------------
Attachment: 1513.1.patch
simple change to let hive started script include conf/hive-env.sh. a template is provided as an example. ran all tests on 20 and tested by hand that the inclusion works.
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Attachments: 1513.1.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895698#action_12895698 ]
Edward Capriolo commented on HIVE-1513:
---------------------------------------
Anything you put in the bin/ext is sourced as part of the bootstrap process. Could you do something like bin/ext/mystuff.sh?
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1513:
-----------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.7.0
Resolution: Fixed
Committed. Thanks Joydeep!
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Fix For: 0.7.0
>
> Attachments: 1513.1.patch, 1513.2.patch, hive-1513.3.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895959#action_12895959 ]
Joydeep Sen Sarma commented on HIVE-1513:
-----------------------------------------
yes - it's possible. however a lot of variables etc. are initialized by the time we get to loading ext/*.sh. for example we allow HADOOP_HEAPSIZE to be specified via env var. but aside from doing an export before launching the hive script, there's no way to configure this externally. the ext/* trick wouldn't work cause it's comes too late.
i think this is simple enough - we can just source a conf/hive-env.sh or something of the sort so that admins can provide right values for all these vars based on their requirements via config files.
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896229#action_12896229 ]
Ning Zhang commented on HIVE-1513:
----------------------------------
Looks good in general. One minor question though. Currently the hive cli also set HADOOP_HEAPSIZE and HADOOP_OPTS right before hadoop is called (in bin/ext/util/execHiveCmd.sh). Should we rename conf/hive-env.sh.template to hive-env.sh and move these two parameter there?
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch, 1513.2.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896269#action_12896269 ]
Ning Zhang commented on HIVE-1513:
----------------------------------
+1. will commit if tests pass.
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch, 1513.2.patch, hive-1513.3.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1513) hive starter scripts should load
admin/user supplied script for configurability
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1513:
------------------------------------
Attachment: 1513.2.patch
forgot to add one file
> hive starter scripts should load admin/user supplied script for configurability
> -------------------------------------------------------------------------------
>
> Key: HIVE-1513
> URL: https://issues.apache.org/jira/browse/HIVE-1513
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: CLI
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 1513.1.patch, 1513.2.patch
>
>
> it's difficult to add environment variables to Hive starter scripts except by modifying the scripts directly. this is undesirable (since they are source code). Hive starter scripts should load a admin supplied shell script for configurability. This would be similar to what hadoop does with hadoop-env.sh
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.