You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2010/01/18 21:16:54 UTC
[jira] Created: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
----------------------------------------------------------------------------------------------------------------------------------
Key: CHUKWA-447
URL: https://issues.apache.org/jira/browse/CHUKWA-447
Project: Hadoop Chukwa
Issue Type: Improvement
Components: input tools
Affects Versions: 0.3.0, 0.4.0
Environment: Redhat EL 5.1, Java 6
Reporter: Eric Yang
While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-447:
-----------------------------
Attachment: Picture 4.png
The load one minute average graph for a week. The change was implemented at Jan 16 midnight, and notice the load jumped by 50%.
> Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-447
> URL: https://issues.apache.org/jira/browse/CHUKWA-447
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: input tools
> Affects Versions: 0.3.0, 0.4.0
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Attachments: Picture 4.png
>
>
> While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802347#action_12802347 ]
Eric Yang commented on CHUKWA-447:
----------------------------------
The commands are identical before and after. Here is what is in my initial_adaptors:
{code}
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60 /usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60 /usr/bin/top -b -n 1 -c 0
{code}
> Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-447
> URL: https://issues.apache.org/jira/browse/CHUKWA-447
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: input tools
> Affects Versions: 0.3.0, 0.4.0
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Attachments: Picture 4.png
>
>
> While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801927#action_12801927 ]
Ari Rabkin commented on CHUKWA-447:
-----------------------------------
Interesting. And sort of surprising, since both are spawning processes in a loop. Increased 50% is how large an increase in absolute terms?
> Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-447
> URL: https://issues.apache.org/jira/browse/CHUKWA-447
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: input tools
> Affects Versions: 0.3.0, 0.4.0
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Attachments: Picture 4.png
>
>
> While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801943#action_12801943 ]
Ari Rabkin commented on CHUKWA-447:
-----------------------------------
Can you say anything about the configuration settings in use? Which processes are you forking? Same as before, or different?
> Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-447
> URL: https://issues.apache.org/jira/browse/CHUKWA-447
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: input tools
> Affects Versions: 0.3.0, 0.4.0
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Attachments: Picture 4.png
>
>
> While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-447) Load increased by 50% when using
ExecAdaptor instead of
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801928#action_12801928 ]
Ari Rabkin commented on CHUKWA-447:
-----------------------------------
NM, didn't see graph. Hrm. Let me do some profiling here. But this probably does add more impetus to CHUKWA-419 or equivalent.
> Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-447
> URL: https://issues.apache.org/jira/browse/CHUKWA-447
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: input tools
> Affects Versions: 0.3.0, 0.4.0
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Attachments: Picture 4.png
>
>
> While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.