You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2010/04/06 20:39:33 UTC
[jira] Created: (CHUKWA-472) TsProcessor: make date format
configurable
TsProcessor: make date format configurable
------------------------------------------
Key: CHUKWA-472
URL: https://issues.apache.org/jira/browse/CHUKWA-472
Project: Hadoop Chukwa
Issue Type: Improvement
Reporter: Bill Graham
The TsProcessor's default date format and it's date format for a given data type should both be configurable.
- To set time format for a given data type:
{noformat}
<property>
<name>TsProcessor.time.format.some_data_type</name>
<value>yyyy-MM-dd HH:mm:ss,SSS</value>
</property>
{noformat}
- To set the default time format:
{noformat}
<property>
<name>TsProcessor.default.time.format</name>
<value>yyyy-MM-dd HH:mm:ss,SSS</value>
</property>
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860430#action_12860430 ]
Ari Rabkin commented on CHUKWA-472:
-----------------------------------
+1 will commit this weekend barring objection.
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch, CHUKWA-472.2.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham reassigned CHUKWA-472:
----------------------------------
Assignee: Bill Graham
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-472:
-------------------------------
Status: Patch Available (was: Open)
Release Note: TsProcessor time format is configurable.
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-472:
-------------------------------
Status: Patch Available (was: Open)
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch, CHUKWA-472.2.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-472:
-------------------------------
Attachment: CHUKWA-472.1.patch
Attaching CHUKWA-472.1.patch. This patch requires CHUKWA-471.patch to be applied first.
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-472:
-------------------------------
Attachment: CHUKWA-472.2.patch
Attaching CHUKWA-472.2.patch, which implements the add'l functionality described above.
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch, CHUKWA-472.2.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-472:
-----------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I just committed this, thanks Bill.
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch, CHUKWA-472.2.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861058#action_12861058 ]
Bill Graham commented on CHUKWA-472:
------------------------------------
Thanks Eric.
FYI, for anyone with a tendency to copy-and-paste, the sample configuration regex values shown above should only have single-backslash-escapes when used in configuration files (the double-escapes were taken from values in java code).
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch, CHUKWA-472.2.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-472) TsProcessor: make date format
configurable
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated CHUKWA-472:
-------------------------------
Status: Open (was: Patch Available)
Canceling this patch, since I want to add one more bit of functionality that I think will be useful. The current implementation expects the date to be the first set of characters in the record. In some cases (i.e. Apache logs) that's not the case. Adding the ability to optionally specify a regular expression to locate where the date string exists in the record.
For a record like this for example, you could use configs like below:
{{10.10.182.49 [22/Apr/2010:15:07:27 -0700] "" 200 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "some.site.com:8076"}}
{noformat}
<property>
<name>TsProcessor.time.regex.some_data_type</name>
<value>^(?:[\\d.]+) \\[(\\d{2}/\\w{3}/\\d{4}:\\d{2}:\\d{2}:\\d{2} [-+]\\d{4})\\] .*</value>
</property>
{noformat}
{noformat}
<property>
<name>TsProcessor.default.time.regex</name>
<value>^(?:[\\d.]+) \\[(\\d{2}/\\w{3}/\\d{4}:\\d{2}:\\d{2}:\\d{2} [-+]\\d{4})\\] .*</value>
</property>
{noformat}
> TsProcessor: make date format configurable
> ------------------------------------------
>
> Key: CHUKWA-472
> URL: https://issues.apache.org/jira/browse/CHUKWA-472
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: CHUKWA-472.1.patch
>
>
> The TsProcessor's default date format and it's date format for a given data type should both be configurable.
> - To set time format for a given data type:
> {noformat}
> <property>
> <name>TsProcessor.time.format.some_data_type</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
> - To set the default time format:
> {noformat}
> <property>
> <name>TsProcessor.default.time.format</name>
> <value>yyyy-MM-dd HH:mm:ss,SSS</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.