You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2008/10/16 23:30:46 UTC
[jira] Created: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Improve data loader for collecting metrics and log files from hadoop and system
-------------------------------------------------------------------------------
Key: HADOOP-4433
URL: https://issues.apache.org/jira/browse/HADOOP-4433
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/chukwa
Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
Reporter: Eric Yang
Assignee: Eric Yang
Enhancement should include:
- Ability to stream /var/log/messages
- Better handling of startup and shutdown chukwa processes
- Ability to monitor torque/hod allocated clusters
- Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4433:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Please make much smaller patches.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433-2.patch, HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Attachment: HADOOP-4433.patch
- Added startup and shutdown script
- Added torque metrics data loader
- Improve handling of Exec Plugin
- Added Test cases for File Tailing Adaptors
- Added Test cases for Start streaming at specific offset
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640884#action_12640884 ]
Hadoop QA commented on HADOOP-4433:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392397/HADOOP-4433.patch
against trunk revision 705942.
-1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/console
This message is automatically generated.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Attachment: HADOOP-4433-2.patch
- Merge the changes from trunk build.xml for ChukImp Test cases.
- Remove author tag from ChukwaFileAppender class.
Patches will become smaller and smaller, I promise. :)
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433-2.patch, HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Status: Patch Available (was: Open)
Changes from previous patch:
-Resolved trunk build.xml file conflict.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433-2.patch, HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Attachment: (was: HADOOP-4433.patch)
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645244#action_12645244 ]
Hudson commented on HADOOP-4433:
--------------------------------
Integrated in Hadoop-trunk #652 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/652/])
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433-2.patch, HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642208#action_12642208 ]
Jerome Boulon commented on HADOOP-4433:
---------------------------------------
Important patch for Chukwa since it's fixing some blocking bugs at the adaptor level and the ExecPlugin
+1
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640902#action_12640902 ]
Eric Yang commented on HADOOP-4433:
-----------------------------------
False positive on the author tag. The author tag was existed in Log4j DailyRollingFileAppender.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Release Note:
- Added startup and shutdown script
- Added torque metrics data loader
- Improve handling of Exec Plugin
- Added Test cases for File Tailing Adaptors
- Added Test cases for Start streaming at specific offset
Status: Patch Available (was: Open)
This patch depends on the patch for HADOOP-4431.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643327#action_12643327 ]
Ari Rabkin commented on HADOOP-4433:
------------------------------------
Looks okay to me. +1
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433-2.patch, HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4433:
----------------------------------
Status: Open (was: Patch Available)
The patch doesn't apply against trunk, please regenerate it. It fails on build.xml. In the future, please make smaller patches.
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4433) Improve data loader for collecting
metrics and log files from hadoop and system
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-4433:
------------------------------
Attachment: HADOOP-4433.patch
- Added startup and shutdown script
- Added torque metrics data loader
- Improve handling of Exec Plugin
- Added Test cases for File Tailing Adaptors
- Added Test cases for Start streaming at specific offset
> Improve data loader for collecting metrics and log files from hadoop and system
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4433
> URL: https://issues.apache.org/jira/browse/HADOOP-4433
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Environment: Red Hat Enterprise Linux 4.5, 5, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-4433.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Enhancement should include:
> - Ability to stream /var/log/messages
> - Better handling of startup and shutdown chukwa processes
> - Ability to monitor torque/hod allocated clusters
> - Add serialization and deserialization of hadoop metrics to JSON log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.