You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Roman Shaposhnik (Created) (JIRA)" <ji...@apache.org> on 2012/01/27 03:21:40 UTC
[jira] [Created] (HIVE-2757) hive can
hive can
--------
Key: HIVE-2757
URL: https://issues.apache.org/jira/browse/HIVE-2757
Project: Hive
Issue Type: Bug
Reporter: Roman Shaposhnik
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261124#comment-13261124 ]
Carl Steinbach commented on HIVE-2757:
--------------------------------------
bq. This unique feature of Hive makes discovery of Hadoop executor script happen at the level of Java code.
Just to make sure I understand, you're saying that right now the bin/hive shell script calls bin/hadoop, and you want to modify it so that bin/hive bypasses bin/hadoop and instead calls java directly with the appropriate Hive class? That sounds like an improvement to me. Even better would be to eliminate the Hive shell scripts entirely (or at least whittle them down to nothing) and push most of this logic into Java.
Note that we're actually moving in the direction of removing the dependency on bin/hadoop entirely. HIVE-2646 is part of that effort and just got committed.
bq. P.S. I have also took the liberty of removing ConfVars.HADOOPCONF since I don't think it is used anymore.
That looks good to me. I wasn't able to find any references to this variable either.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Buddhika Chamith De Alwis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Buddhika Chamith De Alwis updated HIVE-2757:
--------------------------------------------
Attachment: hive-2757.diff
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262044#comment-13262044 ]
Carl Steinbach commented on HIVE-2757:
--------------------------------------
@Ed: o.a.h.Configuration is responsible for locating the Hadoop configuration files on the classpath at time of initialization. Since HiveConf extends Configuration we inherit this functionality for free. I agree that it may eventually become necessary to dynamically modify the classpath at runtime to include the Hadoop conf directory, but right now we don't do that, and in the meantime ConfVars.HADOOPCONF is dead code (which is really confusing, since developers always assume that the code they read is used somewhere).
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270238#comment-13270238 ]
Phabricator commented on HIVE-2757:
-----------------------------------
ashutoshc has requested changes to the revision "HIVE-2757 [jira] hive can't find hadoop executor scripts without HADOOP_HOME set Submitting this for review on behalf of Roman Shaposhnik based on the patch he uploaded here: https://issues.apache.org/jira/secure/attachment/12525320/HIVE-2757.patch.txt".
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:914 Please use File.Separator instead of "/" to avoid OS-specific code.
REVISION DETAIL
https://reviews.facebook.net/D3075
BRANCH
HIVE-2757-executor-hadoop-home
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263267#comment-13263267 ]
Ashutosh Chauhan commented on HIVE-2757:
----------------------------------------
@Roman,
Your patch is definitely an improvement over status quo. HADOOP_HOME is deprecated and we have to accomodate for HADOOP_PREFIX for Yarn. Instead of doing those changes in bin/hive, it definitely makes sense to do it here. Also, patch doesn't change the existing behavior. So, I am +1 on the approach.
Can you post the full patch so that I can see if there are other changes you have in mind.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261890#comment-13261890 ]
Roman Shaposhnik commented on HIVE-2757:
----------------------------------------
@Edward, not sure I understand -- at this point it is an unused variable and it serves no purpose. If, during the course of development of this patch, we find a use for it -- sure we'll keep it.
Does it make sense?
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273011#comment-13273011 ]
Hudson commented on HIVE-2757:
------------------------------
Integrated in Hive-trunk-h0.21 #1423 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1423/])
HIVE-2757. Hive can't find hadoop executor scripts without HADOOP_HOME set (Roman Shaposhnik via cws) (Revision 1336906)
Result = FAILURE
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336906
Files :
* /hive/trunk/bin/ext/help.sh
* /hive/trunk/bin/hive
* /hive/trunk/bin/init-hive-dfs.sh
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.10.0
>
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Buddhika Chamith De Alwis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265906#comment-13265906 ]
Buddhika Chamith De Alwis commented on HIVE-2757:
-------------------------------------------------
Recently I had this requirement of getting hive running in local mode without the assumption of HADOOP_HOME being set. My use case was to run Hive embedded inside an OSGi environment running the Hive Thrift Service. So I had to make some changes in local mode job submission codes to directly invoke Hadoop instead of going through Hadoop scripts. I am attaching the changes as a patch in case you folks find it useful. I took the liberty of defining couple of new properties so that it would continue to work with Hadoop scripts as normal if above properties are not explicitly set. Anyway this doesn't take away the HADOOP_HOME requirement to start hive using hive scripts since they are heavily dependent on being HADOOP_HOME set. In my case this was sufficient since I was starting Hive server programatically without going through hive scripts. Also here I have used a HADOOPLIB property where I assume all Hadoop related dependencies are present. In my case this was not bound to HADOOP_HOME, was just a specific location inside OSGi container where all Hive and Hadoop jars were present together.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated HIVE-2757:
-----------------------------------
Component/s: CLI
Description:
The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
and pass that as a property into the JVM.
Affects Version/s: 0.8.0
Summary: hive can't find hadoop executor scripts without HADOOP_HOME set (was: hive can)
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262283#comment-13262283 ]
Edward Capriolo commented on HIVE-2757:
---------------------------------------
We are missing the larger issue here. Bigtop only cares about two entry points does not mean there are only two we should care about. We have hive web interface, hive-thrift-service, lineage tool. Oozie should probably interface with the Hive-thrift-service I did that here: https://github.com/edwardcapriolo/m6d_oozie/blob/master/src/main/java/com/m6d/oozie/HiveServiceBAction.java. Thrift is the best programmatic way to interface with hive.
We have a long standing ticket open to standardize the environment for all these processes so they all have a common entry point. (I can not find the ticket ATM).
We are always going to need something like HADOOP_HOME because hive assumes hadoop jars are "provided" (like in maven terms) and hadoop configuration is "provided". Unless we copy all the hadoop jars to hive/lib and copy all the hadoop configuration to hive. With the patch about we still use HADOOP_HOME to build the classpath to start hive.
Also HADOOP_HOME is deprecated but in the setup docs "http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/SingleCluster.html"
"Assuming that the environment variables $HADOOP_COMMON_HOME, $HADOOP_HDFS_HOME, $HADOO_MAPRED_HOME, $YARN_HOME, $JAVA_HOME and $HADOOP_CONF_DIR have been set appropriately. Set $$YARN_CONF_DIR the same as $HADOOP_CONF_DIR"
I still feel like hive is going to end up using environment variables to start up.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276645#comment-13276645 ]
Hudson commented on HIVE-2757:
------------------------------
Integrated in Hive-trunk-h0.21 #1434 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1434/])
HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757
(Zhenxiao Luo via Carl Steinbach)
Summary: HIVE-3014: Fix metastore test failures caused by HIVE-2757
Test Plan: EMPTY
Reviewers: JIRA, cwsteinbach
Reviewed By: cwsteinbach
Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004)
Result = SUCCESS
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.10.0
>
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261921#comment-13261921 ]
Roman Shaposhnik commented on HIVE-2757:
----------------------------------------
@Carl, bin/hive shell script calling bin/hadoop is a bit of an orthogonal issue here I think. The basic problem I'm trying to address in this JIRA is this: hive has at least 2 entry points Bigtop cares about for integration purposes:
# bin/hive
# org.apache.hadoop.hive.cli.CliDriver
Regardless of which one is used, though, hive Java code will end up exec'ing hadoop launcher script when it comes to job submission. The reason I'm brining up the 2 is simple: given that #2 exists and is used by things like Oozie we can't rely on shell-level computation of environment (like finding the hadoop executable and passing that to java code via a property, etc.).
What this JIRA is trying to accomplish is to push the logic of finding hadoop executable script (if none is given) back to the java code, since because of #2 it is the only place where it can be done reliably.
Makes sense?
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated HIVE-2757:
-----------------------------------
Status: Patch Available (was: Open)
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2757:
------------------------------
Attachment: HIVE-2757.D3075.1.patch
cwsteinbach requested code review of "HIVE-2757 [jira] hive can't find hadoop executor scripts without HADOOP_HOME set Submitting this for review on behalf of Roman Shaposhnik based on the patch he uploaded here: https://issues.apache.org/jira/secure/attachment/12525320/HIVE-2757.patch.txt".
Reviewers: JIRA
HIVE-2757. hive can't find hadoop executor scripts without HADOOP_HOME set
The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it
would be really nice if bin/hive can be modified to capture the which hadoop
and pass that as a property into the JVM.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D3075
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/7011/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266715#comment-13266715 ]
Roman Shaposhnik commented on HIVE-2757:
----------------------------------------
@Ashutosh,
I think at this point I'd go for the minimally invasive patch (which is attached). Potential improvements would include merging the stuff that Buddhika posted (and better yet, following up with a patch that migrates us to ProcessBuilder). However, I believe all of those things need to be handled in separate JIRAs.
@Edward,
I agree with your comments on standardizing the environment. It is fine for Hive to keep depending on env. variables, but it also needs to have reasonable defaults.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261876#comment-13261876 ]
Edward Capriolo commented on HIVE-2757:
---------------------------------------
I am -1 on removing ConfVars.HADOOPCONF, because when we do remove HADOOP_BIN hive is going to need some way to locate the hadoop configuration files. Then this variable is useful again.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach reassigned HIVE-2757:
------------------------------------
Assignee: Roman Shaposhnik
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.10.0
>
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated HIVE-2757:
-----------------------------------
Attachment: HIVE-2757.patch.txt
Here's an example patch that is not meant for inclusion, but rather to generate discussion whether such an approach would be acceptable.
Basically, the fundamental problem is that Hive java code can be used in other projects (like Oozie) and hence it can't rely on launcher shell scripts always passing correct set of properties along based on querying the environment at shell level.
This unique feature of Hive makes discovery of Hadoop executor script happen at the level of Java code. The patch contains a very naive attempt at doing that while maintaining backward compatibility with Hadoop 0.20.X and older releases. The most notable feature that is still missing is an ability to discover Hadoop that's part of the user's PATH. Before I implement that, however, I'd like to ask whether exec'ing via ProcessBuilder won't be a better option, rather than me manually tying to parse PATH (error prone).
Please let me know what you think.
P.S. I have also took the liberty of removing ConfVars.HADOOPCONF since I don't think it is used anymore.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2757:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.10.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Roman!
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.10.0
>
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271820#comment-13271820 ]
Roman Shaposhnik commented on HIVE-2757:
----------------------------------------
Attaching a patch that incorporated all the feedback from the reviews and also got tested. Attaching here since FB is broken.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated HIVE-2757:
-----------------------------------
Attachment: HIVE-2757.patch.txt
Sorry for the delay -- it seems I don't have much luck with running Hive unit tests on my machine :-(
On the positive side -- I tested this patch against a real cluster and it works as expected.
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266866#comment-13266866 ]
Carl Steinbach commented on HIVE-2757:
--------------------------------------
@Roman: Can you submit a phabricator review request? Thanks.
https://cwiki.apache.org/Hive/phabricatorcodereview.html
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2757) hive can't find hadoop executor
scripts without HADOOP_HOME set
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated HIVE-2757:
-----------------------------------
Attachment: HIVE-2757-2.patch.txt
> hive can't find hadoop executor scripts without HADOOP_HOME set
> ---------------------------------------------------------------
>
> Key: HIVE-2757
> URL: https://issues.apache.org/jira/browse/HIVE-2757
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Roman Shaposhnik
> Attachments: HIVE-2757-2.patch.txt, HIVE-2757.D3075.1.patch, HIVE-2757.patch.txt, HIVE-2757.patch.txt, hive-2757.diff
>
>
> The trouble is that in Hadoop 0.23 HADOOP_HOME has been deprecated. I think it would be really nice if bin/hive can be modified to capture the which hadoop
> and pass that as a property into the JVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira