[jira] [Commented] (CHUKWA-700) Revisit Chukwa metrics schema design for HBase

["shreyas subramanya (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CHUKWA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816928#comment-13816928 ] 

shreyas subramanya commented on CHUKWA-700:
-------------------------------------------

I think we need to provide support for multiple custom schema, since the performance would depend on the actual scan scenario (as observed in Chukwa-667). Just like we can configure the hbase table and column family in demux processor, we need to provide annotation for OutputCollector, so that the demux processor can dictate the hbase schema.

> Revisit Chukwa metrics schema design for HBase
> ----------------------------------------------
>
>                 Key: CHUKWA-700
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-700
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Collection
>    Affects Versions: 0.6.0
>         Environment: MacOSX, Java
>            Reporter: Eric Yang
>
> Current Chukwa HBase schema looks like this:
> {code}
> <timestamp>-<primaryKey>   <columnFamily>:<cell>...
> {code}
> Monotonic increasing timestamp can not evenly distribute across region servers without special handle and care periodically.
> It is time to revise the schema, and proposed schema looks like this:
> {code}
> <hhddmmyyyy>-<primaryId>  cf:<cell>...
> {code}
> Timestamp is stored with cell, row key helps to split data by hour, and a full hour of metrics is stored on the same row.  PrimaryKey is replaced with hash id of the primary key.  Metrics tables to aggregate metrics:
> chukwaMetrics -> chukwaMetricsMonthly -> chukwaMetricsYearly



--
This message was sent by Atlassian JIRA
(v6.1#6144)