You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2010/10/03 19:43:32 UTC
[jira] Created: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Syslog Adaptor for mapping Syslog facility name to Chukwa data type
-------------------------------------------------------------------
Key: CHUKWA-530
URL: https://issues.apache.org/jira/browse/CHUKWA-530
Project: Chukwa
Issue Type: New Feature
Components: data collection
Environment: Java 6, Mac OS X
Reporter: Eric Yang
Assignee: Eric Yang
UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921466#action_12921466 ]
Ari Rabkin commented on CHUKWA-530:
-----------------------------------
+1 looks good.
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-530-draft.patch, CHUKWA-530.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917370#action_12917370 ]
Eric Yang commented on CHUKWA-530:
----------------------------------
I am thinking to add more configure parameters to Chukwa Agent config file. i.e.:
Trigger for adding syslog adaptor
{noformat}
add SyslogAdaptor SYSLOG 9095 0
{noformat}
SyslogAdaptor would look up:
{noformat}
<property>
<name>syslog.adaptor.[port].[facility name]</name>
<value>[chukwa data type]</value>
</property>
{noformat}
Example:
{noformat}
<property>
<name>syslog.adaptor.9095.LOCAL0</name>
<value>HADOOP</value>
</property>
<property>
<name>syslog.adaptor.9095.LOCAL1</name>
<value>HadoopMetrics</value>
</property>
<property>
<name>syslog.adaptor.9095.AUDIT</name>
<value>AUDITLOG</value>
</property>
{noformat}
This means each udp port can host up to 24 data types. Make sense?
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-530-draft.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-530:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks Ari, I just committed this.
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.5.0
>
> Attachments: CHUKWA-530-draft.patch, CHUKWA-530.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-530:
-----------------------------
Status: Patch Available (was: Open)
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-530-draft.patch, CHUKWA-530.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-530:
-----------------------------
Attachment: CHUKWA-530.patch
Revised patch to include javadoc and test case. New feature includes mapping of facility name to Chukwa data type.
Usage in Chukwa Agent configuration file:
{noformat}
<property>
<name>syslog.adaptor.port.9095.facility.LOCAL1</name>
<value>HADOOP</value>
</property>
{noformat}
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-530-draft.patch, CHUKWA-530.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-530) Syslog Adaptor for mapping Syslog
facility name to Chukwa data type
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-530:
-----------------------------
Attachment: CHUKWA-530-draft.patch
Working implementation.
> Syslog Adaptor for mapping Syslog facility name to Chukwa data type
> -------------------------------------------------------------------
>
> Key: CHUKWA-530
> URL: https://issues.apache.org/jira/browse/CHUKWA-530
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Environment: Java 6, Mac OS X
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-530-draft.patch
>
>
> UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.
> My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.