You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Robert Surówka (Created JIRA)" <ji...@apache.org> on 2011/10/12 10:31:11 UTC
[jira] [Created] (HIVE-2500) Allow Hive to be debugged remotely
Allow Hive to be debugged remotely
----------------------------------
Key: HIVE-2500
URL: https://issues.apache.org/jira/browse/HIVE-2500
Project: Hive
Issue Type: New Feature
Reporter: Robert Surówka
Assignee: Robert Surówka
Priority: Minor
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125672#comment-13125672 ]
Robert Surówka commented on HIVE-2500:
--------------------------------------
The review request at https://reviews.apache.org/r/2354/
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Ning Zhang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139675#comment-13139675 ]
Ning Zhang commented on HIVE-2500:
----------------------------------
+1. Will commit if tests pass.
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch, HIVE-2500.4.patch, HIVE-2500.5.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Attachment: HIVE-2500.5.patch
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch, HIVE-2500.4.patch, HIVE-2500.5.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Status: Patch Available (was: Open)
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Attachment: HIVE-2500.3.patch
The changes with this patch are as follows (I made some design changes to the comment from OCT 14) :
-now the child JVMs should work correctly with debug (user decides whether they should be also started in debug mode or not).
-When child JVM starts in debug mode it will start listening on some random unused TCP port.
-By default child JVMs start in debug mode, but don't wait with execution for the debugger to connect (yet if someone would later decide that he actually wants to connect to any of them, he will be able to do that).
-user can specify debug options, as described in debug_help()
-I am very unproficient in bash, so if there are any changes you would like to see to make that code look better please let me know.
-Most ugly part of this patch are probably the regex expressions/string operations added to MapRedTask. They could be made significantly simpler, if additional environment variables were exported, and totally removed if small changes to hadoop starting scripts would be made. Yet I decided (maybe wrongly) that it's best to close all that entropy in configureDebugVariablesForChildJVM method instead of spilling it around (especially that it's just for debug).
The current diff is at: https://reviews.apache.org/r/2354/diff/3/
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Attachment: HIVE-2500.1.patch
To connect to debug one can use e.g. Eclipse (Debug configurations -> Remote Java Application (New) -> (Standard (Socket Attach) | host where Hive runs (e.g localhost) | port 8000 | it's good to allow remote VM termination -> Debug.
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Attachment: HIVE-2500.2.patch
Thank you for this advice, I updated the diff (and review request) accordingly.
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127899#comment-13127899 ]
Robert Surówka commented on HIVE-2500:
--------------------------------------
Sorry, I meant MapRedTask.
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139903#comment-13139903 ]
Hudson commented on HIVE-2500:
------------------------------
Integrated in Hive-trunk-h0.21 #1046 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1046/])
HIVE-2500. Allow Hive to be debugged remotely (Robert Surowka via Ning Zhang)
nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195288
Files :
* /hive/trunk/bin/ext/debug.sh
* /hive/trunk/bin/ext/help.sh
* /hive/trunk/bin/hive
* /hive/trunk/conf/hive-env.sh.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch, HIVE-2500.4.patch, HIVE-2500.5.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Status: Patch Available (was: Open)
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Status: Open (was: Patch Available)
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Ning Zhang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-2500:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Robert!
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch, HIVE-2500.4.patch, HIVE-2500.5.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2500:
---------------------------------
Attachment: HIVE-2500.4.patch
Reintroduced the change to the hive-env.sh.template that I missed in the last patch.
Note: due to the change I was requested to make, to add debug options to HADOOP_CLIENT_OPTS instead of HADOOP_OPTS, the debugging won't work unless at some point HADOOP_CLIENT_OPTS will be introduced into HADOOP_OPTS (or will be passed to JVM directly). This could be possibly best done by adding line like:
export HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" to hadoop-env.sh
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch, HIVE-2500.3.patch, HIVE-2500.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Robert Surówka (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127896#comment-13127896 ]
Robert Surówka commented on HIVE-2500:
--------------------------------------
Apparently this diff is still wrong (I will have to do small fix to MapReduceTask.java, when debug off, it will add just one if check for a key being in a map per execute() call of that task, and only if new JVM is about to be started).
As I am working on it I plan to add following features:
-user could specify port on which Hive JVM will wait (8000 will be default).
-user could specify whether child JVMs should be also started in debug mode (default: not). If started in debug mode, child JVMs will wait on a different ports than parent (and I will try to make it in such a way, so probability of ports collision would be very small (to make it impossible one would need to use some between process communication, since children can start their own children, and I don't want to do that yet (especially that HIVE may be deployed on various os-es). User will know on which port new JVMs are waiting, since upon start they print that information. New JVMs will be started on ports higher than port of the initial JVM - so if someone needs, he can set initial port to be higher than any other already used TCP port).
ETA: by mid of next week (hopefully)
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125938#comment-13125938 ]
Ashutosh Chauhan commented on HIVE-2500:
----------------------------------------
Useful debugging feature. Robert, Instead of HADOOP_OPTS we should export HADOOP_CLIENT_OPTS in such cases otherwise we may run into problems such as MAPREDUCE-3112
> Allow Hive to be debugged remotely
> ----------------------------------
>
> Key: HIVE-2500
> URL: https://issues.apache.org/jira/browse/HIVE-2500
> Project: Hive
> Issue Type: New Feature
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2500.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira