You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2012/04/27 00:18:49 UTC
[jira] [Created] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Bikas Saha created MAPREDUCE-4203:
-------------------------------------
Summary: Create equivalent of ProcfsBasedProcessTree for Windows
Key: MAPREDUCE-4203
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4203:
----------------------------------
Attachment: MAPREDUCE-4203.branch-1-win.3.patch
Attaching patch after review comments.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397763#comment-13397763 ]
Chuan Liu commented on MAPREDUCE-4203:
--------------------------------------
+1 Thanks!
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4203:
----------------------------------
Attachment: MAPREDUCE-4203.branch-1-win.2.patch
Fix some bugs in formatting.
TestTaskTrackerMemoryManager now passes on Windows and tests the feature functionally.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397199#comment-13397199 ]
Chuan Liu commented on MAPREDUCE-4203:
--------------------------------------
[~bikassaha], change looks good! I just have following two minor suggestions for 'SystemInfo.c' based on branch-1-win.3.patch.
# Use fwprintf() instead of _ftprintf_s(). Because we have UNICODE defined, they should be the same. It is just fwprintf() makes things more clear in my opinion.
# ReportErrorCode() was designed to be used with system error codes. So it would be better to just print an custom error message instead of calling it with EXIT_FAILURE as follows.
{code}
ReportErrorCode(L"CallNtPowerInformation", EXIT_FAILURE);
{code}
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396132#comment-13396132 ]
Jonathan Eagles commented on MAPREDUCE-4203:
--------------------------------------------
Thanks, Bikas. Just trying to prevent Hadoop code from being contaminated with GPL or proprietary code licenses. Sounds like you are already controlling for that.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397766#comment-13397766 ]
Ivan Mitic commented on MAPREDUCE-4203:
---------------------------------------
+1, change looks good, thanks Bikas.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397063#comment-13397063 ]
Bikas Saha commented on MAPREDUCE-4203:
---------------------------------------
ProcfsBasedProcessTree uses procs.getValue().updateAge(oldInfo) to update the age. updateAge() explicitly sets age = oldAge+1. So there is no double increment.
Ah.. sorry. I confused main() with Task(). There is no wrapper around SystemInfo() like Task(). Will add the error mesg.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397221#comment-13397221 ]
Bikas Saha commented on MAPREDUCE-4203:
---------------------------------------
1) I would prefer the secure and generic versions unless you have a strong preference for the wide char version :)
2) I re-used ReportErrorCode so that I dont have to guess where the error reporting is going - stderr or some log file. So I used the standard ReportErrorCode() function that takes care of it globally.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Mahadevan Venkatraman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270004#comment-13270004 ]
Mahadevan Venkatraman commented on MAPREDUCE-4203:
--------------------------------------------------
This is great.
Could you propagate the error code(s) from GetLastError() call for the printProcessTree() and printSystemInfo() in the error paths?
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Chuan Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397252#comment-13397252 ]
Chuan Liu commented on MAPREDUCE-4203:
--------------------------------------
# {quote}
I would prefer the secure and generic versions unless you have a strong preference for the wide char version
{quote}
You can use fwprintf_s(). I am fine if you want to stay with tchar version. It is just implementation wise, tchar functions are always mapped to either Unicode/wchar version or ANSI version. E.g. we have the following definition in "tchar.h". Because we have explicitly indicate we only support Unicode. I think it more clear if we always use wide char version of the functions. Either way, since they are equivalent, it is your call.
{code}
#define _ftprintf_s fwprintf_s
{code}
# For ReportErrorCode(), it is because we are calling FormatMessageW() inside the function to retrieval the [system error code|http://msdn.microsoft.com/en-us/library/windows/desktop/ms681382.aspx] message inside the function. If you pass in EXIT_FAILURE (1), it will print error message of ERROR_INVALID_FUNCTION (1).
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397670#comment-13397670 ]
Bikas Saha commented on MAPREDUCE-4203:
---------------------------------------
ok. for this specific case I will just print an error message inline.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396594#comment-13396594 ]
Ivan Mitic commented on MAPREDUCE-4203:
---------------------------------------
Change looks good Bikas!
I have some minor comments/questions/suggestions:
1. WindowsBasedProcessTree.java:114: Should the new age be oldAge+1 same as in ProfsBasedProcessTree?
2. LinuxResourceCalculatorPlugin.java:399: Do we need this code: {code}
if(processPid == null) {
// process pid not set. try to obtain on our own
processPid = System.getenv().get("JVM_PID");
}
{code} processPid should have been initialized. If not initialized in all cases, we might have a problem on Windows. To harden the code, might make sense to add a null check for processPid in {{WindowsResourceCalculatorPlugin#getProcResourceValues}} and log an error if processPid is undefined.
3. Should {{ResourceCalculatorPlugin#getResourceCalculatorPlugin()}} fallback to {{WindowsResourceCalculatorPlugin}} on Windows?
4. task.c:335 From documentation, it seems that PagefileUsage is always zero on Windows Server 2008, use PrivateUsage instead?
5. main.c:100: Would you mind updating {{Usage()}} function with information on "systeminfo"? Please include information on the format of the output, as it’s hard to figure out. Same comment for {{task.c!TaskUsage()}}
6. systeminfo.c: Can you please print error info via {{ReportErrorCode()}} when calls into Windows APIs fail?
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393560#comment-13393560 ]
Jonathan Eagles commented on MAPREDUCE-4203:
--------------------------------------------
Bikas, when you say based on winutils, it makes me think you are basing this patch on some existing code. If so, can you comment on the source and licensing.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396046#comment-13396046 ]
Bikas Saha commented on MAPREDUCE-4203:
---------------------------------------
Sorry, I should have been more clear. winutils is a native executable that gets built on Windows. The source for that got added in HADOOP-8235. It provides key functionality like permissions, groups, hard links etc that are required to run Hadoop on Windows. This patch adds more support by adding functionality to winutils to monitor memory etc for tasks.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4203:
----------------------------------
Attachment: MAPREDUCE-4203.branch-1-win.4.patch
Attaching new patch with error mesg change and wide char usage.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397037#comment-13397037 ]
Ivan Mitic commented on MAPREDUCE-4203:
---------------------------------------
bq. ProcessInfo.age initializes to 1. So newpInfo.age += oldpInfo.age does what you say apart from allowing newpInfo to have a different age too.
I see. It seems that they are over-incrementing the value in {{ProcfsBasedProcessTree}} as the default value is also one.
bq. The main calling function will do that with the return code of this function.
I could not find this...would you mind checking?
Thanks!
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4203:
----------------------------------
Attachment: MAPREDUCE-4203.patch
test.cpp
Initial patch with refactorings/new Windows class/tests and native windows code to get resource information.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-4203.
--------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
Release Note: Added an implementation of the process tree for Windows.
I just committed this. Thanks Bikas!
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Fix For: 1-win
>
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4203:
----------------------------------
Attachment: MAPREDUCE-4203.branch-1-win.1.patch
Attaching patch based on winutils.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397001#comment-13397001 ]
Bikas Saha commented on MAPREDUCE-4203:
---------------------------------------
1) ProcessInfo.age initializes to 1. So newpInfo.age += oldpInfo.age does what you say apart from allowing newpInfo to have a different age too.
2) In the real use case, setProcessPid() is called to set the pid. But there are a bunch of gridmix/ tests that depend on the old behavior of snooping the JVM_PID from the env var on the fly and I did not want to change all of them. So I added this to keep the tests passing without change.
3) Forgot to add that. Will do. Thanks!
4) Thats strange because I can see that the values are not zero and in fact similar to values shown in TaskManager. Will change to PrivateUsage anyways to be safe.
5) Will do.
6) The main calling function will do that with the return code of this function.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of
ProcfsBasedProcessTree for Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397288#comment-13397288 ]
Ivan Mitic commented on MAPREDUCE-4203:
---------------------------------------
Thanks for addressing the comments Bikas.
bq. 2.For ReportErrorCode(), it is because we are calling FormatMessageW() inside the function to retrieval the system error code message inside the function. If you pass in EXIT_FAILURE (1), it will print error message of ERROR_INVALID_FUNCTION (1).
Ah, the issue here is that {{CallNtPowerInformation}} returns {{NTSTATUS}} instead of {{WINERROR}} code. We would need to define a new {{ReportNtErrorCode()}} function that would be able to translate the {{NTSTATUS}} code into the message. The implementation would be similar to {{ReportErrorCode()}}, check out this [MSDN link|http://support.microsoft.com/kb/259693]. I don't mind if you'd like to have a separate Jira on this.
> Create equivalent of ProcfsBasedProcessTree for Windows
> -------------------------------------------------------
>
> Key: MAPREDUCE-4203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.patch, test.cpp
>
>
> ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira