You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kanna Karanam (JIRA)" <ji...@apache.org> on 2012/06/13 19:53:42 UTC
[jira] [Created] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Kanna Karanam created HIVE-3127:
-----------------------------------
Summary: Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Key: HIVE-3127
URL: https://issues.apache.org/jira/browse/HIVE-3127
Project: Hive
Issue Type: Bug
Components: Configuration
Affects Versions: 0.9.0, 0.10.0, 0.9.1
Reporter: Kanna Karanam
Fix For: 0.10.0
The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Pass hconf values as XML instead of
command line arguments to child JVM
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo updated HIVE-3127:
----------------------------------
Summary: Pass hconf values as XML instead of command line arguments to child JVM (was: Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows)
> Pass hconf values as XML instead of command line arguments to child JVM
> -----------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kanna Karanam updated HIVE-3127:
--------------------------------
Attachment: HIVE-3127.2.patch.txt
Hive-3127.2.patch.txt – Removed the arguments option completely so have the same logic for Windows & Linux.
Unit Tests – Existing units will cover this scenarios completely.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398576#comment-13398576 ]
Kanna Karanam commented on HIVE-3127:
-------------------------------------
Thanks Carl. Submittied the code review.
https://reviews.apache.org/r/5491/
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399767#comment-13399767 ]
Edward Capriolo commented on HIVE-3127:
---------------------------------------
Welcome to hive :) I am willing to bet we only hit the block merge if the input is larger then hdfs blocks size, or the defaults is high and it normally does not happen during the course of the unit tests.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393576#comment-13393576 ]
Edward Capriolo commented on HIVE-3127:
---------------------------------------
Another interesting piece. If we can write the entire configuration to a file it might make more sense to deprecate the passing individual elements entirely.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3127:
---------------------------------
Component/s: Windows
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396419#comment-13396419 ]
Edward Capriolo commented on HIVE-3127:
---------------------------------------
I vote we remove the arguments options entirely. Unix/Linux has the same max command line argument issue although people hit it less frequently because it is larger. We do not need two code paths and the performance of writing a file is nothing compared to overall mr time.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400949#comment-13400949 ]
Edward Capriolo commented on HIVE-3127:
---------------------------------------
So I have found a reference to BlockMergeTask in ./ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java .
Also the last release note was. ./build/dist/RELEASE_NOTES.txt: * [HIVE-2413] - BlockMergeTask ignores client-specified jars
vi ./ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
You could be correct in that the main is unreachable now. Maybe someone added it as a hack for direct testing, it would be good to cut-it from the code if it is not reachable. Maybe one of the other devs will chime in.
If not one else has any insight I will review and +1 tomorrow.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399754#comment-13399754 ]
Kanna Karanam commented on HIVE-3127:
-------------------------------------
Thanks Edward, I will look into it. I wonder how I got 100% test pass. It looks like there are no unit tests to test this scenario or they are passing with default conf values.
Can you please suggest a query to create a unit test for this?
Thanks
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kanna Karanam updated HIVE-3127:
--------------------------------
Attachment: HIVE-3127.3.patch.txt
Updated the patch with modifications to BlockMergeTask.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-3127:
-----------------------------------
Assignee: Kanna Karanam
Status: Patch Available (was: Open)
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393667#comment-13393667 ]
Kanna Karanam commented on HIVE-3127:
-------------------------------------
Thanks Edward, I thought of removing the arguments option completely, but I felt that it can introduce a small perf hit on Linux unnecessarily as we are creating/deleting a temp files for every task execution. If we feel that this perf hit is negligible then I will remove the arguments option.
Thanks
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kanna Karanam updated HIVE-3127:
--------------------------------
Attachment: HIVE-3127.1.patch.txt
Attached the patch
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Pass hconf values as XML instead of
command line arguments to child JVM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402884#comment-13402884 ]
Hudson commented on HIVE-3127:
------------------------------
Integrated in Hive-trunk-h0.21 #1517 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1517/])
HIVE-3127 Pass hconf values as XML instead of command line arguments to child JVM. Kanna Karanam (via egc) (Revision 1354781)
Result = FAILURE
ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354781
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
> Pass hconf values as XML instead of command line arguments to child JVM
> -----------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399743#comment-13399743 ]
Edward Capriolo commented on HIVE-3127:
---------------------------------------
Kanna,
Your missing something. The BlockMergeTask ask uses -jobconf . You will need to patch that as well.
[edward@tablitha trunk]$ grep -R '\-jobconf' ./*
{noformat}
./ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java: } else if (args[i].equals("-jobconf")) {
{noformat}
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3127) Pass hconf values as XML instead of
command line arguments to child JVM
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo updated HIVE-3127:
----------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thank you.
> Pass hconf values as XML instead of command line arguments to child JVM
> -----------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400112#comment-13400112 ]
Kanna Karanam commented on HIVE-3127:
-------------------------------------
@Edward - Thanks for helping me understand the hive better. I couldn’t figure out the caller of for BlockMergeTask::main anywhere in the code. Thanks.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Kanna Karanam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401995#comment-13401995 ]
Kanna Karanam commented on HIVE-3127:
-------------------------------------
Thanks Edward.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Assignee: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt, HIVE-3127.3.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398301#comment-13398301 ]
Carl Steinbach commented on HIVE-3127:
--------------------------------------
@Kanna: please post a review request on reviews.apache.org. Thanks.
> Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Windows
> Affects Versions: 0.9.0, 0.10.0, 0.9.1
> Reporter: Kanna Karanam
> Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). This limit will be exceeded easily when it appends individual –hconf values to the command string. To work around this problem, Write all changed hconf values to a temp file and pass the temp file path to the child jvm to read and initialize the -hconf parameters from file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira