You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2014/10/24 07:16:34 UTC
[jira] [Updated] (TEZ-1698) Use ResourceCalculatorPlugin instead of
ResourceCalculatorProcessTree in Tez
[ https://issues.apache.org/jira/browse/TEZ-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gopal V updated TEZ-1698:
-------------------------
Attachment: ProcfsBasedProcessTree.png
> Use ResourceCalculatorPlugin instead of ResourceCalculatorProcessTree in Tez
> ----------------------------------------------------------------------------
>
> Key: TEZ-1698
> URL: https://issues.apache.org/jira/browse/TEZ-1698
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.2
> Reporter: Gopal V
> Attachments: ProcfsBasedProcessTree.png
>
>
> ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of the current task's process group.
> This is mostly wasted in Tez, since unlike YARN which has to do this since it has the PID for the container-executor process (bash) and has to trace the bash -> java spawn inheritance.
> !ProcfsBasedProcessTree.png!
> The effect of this is less clearly visible with the profiler turned on as this is primarily related to Syscall overhead in the kernel (via the following codepath in YARN).
> {code}
> private List<String> getProcessList() {
> String[] processDirs = (new File(procfsDir)).list();
> ...
> for (String dir : processDirs) {
> try {
> if ((new File(procfsDir, dir)).isDirectory()) {
> processList.add(dir);
> }
> ...
> public void updateProcessTree() {
> if (!pid.equals(deadPid)) {
> // Get the list of processes
> List<String> processList = getProcessList();
> ...
> for (String proc : processList) {
> // Get information for each process
> ProcessInfo pInfo = new ProcessInfo(proc);
> if (constructProcessInfo(pInfo, procfsDir) != null) {
> allProcessInfo.put(proc, pInfo);
> if (proc.equals(this.pid)) {
> me = pInfo; // cache 'me'
> processTree.put(proc, pInfo);
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)