You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Denny Ye (JIRA)" <ji...@apache.org> on 2012/08/02 11:37:02 UTC
[jira] [Created] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Denny Ye created FLUME-1419:
-------------------------------
Summary: Using system time if 'timestamp' property is absent in event header
Key: FLUME-1419
URL: https://issues.apache.org/jira/browse/FLUME-1419
Project: Flume
Issue Type: Improvement
Components: Sinks+Sources
Affects Versions: v1.2.0
Reporter: Denny Ye
Assignee: Denny Ye
Fix For: v1.3.0
If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
Use the local system time to replace it.
java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Unknown Source)
at java.lang.Long.valueOf(Unknown Source)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Posted by "Denny Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denny Ye updated FLUME-1419:
----------------------------
Attachment: FLUME-1418.patch
> Using system time if 'timestamp' property is absent in event header
> -------------------------------------------------------------------
>
> Key: FLUME-1419
> URL: https://issues.apache.org/jira/browse/FLUME-1419
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1418.patch
>
>
> If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
> Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
> Use the local system time to replace it.
> java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
> at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Unknown Source)
> at java.lang.Long.valueOf(Unknown Source)
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Posted by "Denny Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denny Ye updated FLUME-1419:
----------------------------
Attachment: (was: FLUME-1418.patch)
> Using system time if 'timestamp' property is absent in event header
> -------------------------------------------------------------------
>
> Key: FLUME-1419
> URL: https://issues.apache.org/jira/browse/FLUME-1419
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1419.patch
>
>
> If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
> Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
> Use the local system time to replace it.
> java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
> at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Unknown Source)
> at java.lang.Long.valueOf(Unknown Source)
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437603#comment-13437603 ]
Mike Percy commented on FLUME-1419:
-----------------------------------
The functionality here is already available using the timestamp interceptor, and this changes the semantics so that the bucketing is not consistent: whereas right now the logic is event centric, it now can be local time centric, but that depends on whether the event is annotated with timestamp or not.
This change seems like it would make things more confusing and less predictable.
> Using system time if 'timestamp' property is absent in event header
> -------------------------------------------------------------------
>
> Key: FLUME-1419
> URL: https://issues.apache.org/jira/browse/FLUME-1419
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1419.patch
>
>
> If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
> Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
> Use the local system time to replace it.
> java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
> at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Unknown Source)
> at java.lang.Long.valueOf(Unknown Source)
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Posted by "Denny Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denny Ye updated FLUME-1419:
----------------------------
Attachment: FLUME-1419.patch
> Using system time if 'timestamp' property is absent in event header
> -------------------------------------------------------------------
>
> Key: FLUME-1419
> URL: https://issues.apache.org/jira/browse/FLUME-1419
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1419.patch
>
>
> If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
> Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
> Use the local system time to replace it.
> java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
> at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Unknown Source)
> at java.lang.Long.valueOf(Unknown Source)
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1419) Using system time if 'timestamp'
property is absent in event header
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437447#comment-13437447 ]
Hari Shreedharan commented on FLUME-1419:
-----------------------------------------
I am not sure if this is a right thing to do or not. Usually this timestamp's use would be to exactly record the time at which the event was generated, and store it in HDFS based on this information. This exception allows the user to know that he needs to use the interceptor at the first hop, to know when the event was generated and insert the data into the correct file.
I am not sure if this should go in or not, so I am going to wait for other developers to comment.
Thanks!
> Using system time if 'timestamp' property is absent in event header
> -------------------------------------------------------------------
>
> Key: FLUME-1419
> URL: https://issues.apache.org/jira/browse/FLUME-1419
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1419.patch
>
>
> If we want to use pattern for generating HDFS path but the 'timestamp' property is absent in event header, it may cause the exception...
> Event may comes from several Sources and doesn't use Interceptor(pupulate 'timestamp' property). Thus, event may be haven't this property normally.
> Use the local system time to replace it.
> java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:149)
> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:318)
> at org.apache.flume.formatter.output.TestBucketPath.testDateFormatHours(TestBucketPath.java:46)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Unknown Source)
> at java.lang.Long.valueOf(Unknown Source)
> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:147)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira