You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/01/23 20:53:42 UTC
[jira] [Created] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Structured event log for HBase for monitoring and auto-tuning performance
-------------------------------------------------------------------------
Key: HBASE-5262
URL: https://issues.apache.org/jira/browse/HBASE-5262
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192441#comment-13192441 ]
Mikhail Bautin commented on HBASE-5262:
---------------------------------------
I think we could implement something small-scale in HBase itself, so that it can eventually be used for making optimization decisions for compactions and caching. OpenTSDB is probably better for large-scale time series collection. Also, we don't want to introduce a circular dependency between HBase and OpenTSDB.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192504#comment-13192504 ]
Mikhail Bautin commented on HBASE-5262:
---------------------------------------
@Todd: I totally agree. We should also disable event logging for the event log table itself.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192280#comment-13192280 ]
Zhihong Yu commented on HBASE-5262:
-----------------------------------
One aspect we should consider is the role for opentsdb.
Would there be redundancy between the event log table and what opentsdb stores in HBase?
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192179#comment-13192179 ]
Mikhail Bautin commented on HBASE-5262:
---------------------------------------
I was thinking of a lightly-loaded system table different from .META.. This way it is easier to control this feature, and we can even make it optional based on the configuration. Storing event log data in HBase makes sense to me because we don't expect a huge number of records compared to the user's data itself, and we probably don't want to implement another logging framework with HDFS file management, cleaning up old files, etc. The multiple writers to one file question is also there. If we store this data in HBase, then yes, we can use TTL to set event log retention to e.g. one month or so, or make it configurable.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191933#comment-13191933 ]
stack commented on HBASE-5262:
------------------------------
@Mikhail Sounds good. We'd write to hbase because we can't have many writers write same file in hdfs? This system table would be a new one, not .META.? We'd fill it to the end of time or the TTL on the table would get rid of old records (that'd work).
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191692#comment-13191692 ]
Todd Lipcon commented on HBASE-5262:
------------------------------------
I have some preference against JSON -- IMO JSON isn't "strict" enough, so people tend to break the format over time without a centralized schema definition to remind them that this is a real interface. I would be happier to see either avro or protobuf used. But definitely +1 for something that's consumable by non-Java programs and won't need to be actively upgraded when we add or remove fields.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192603#comment-13192603 ]
stack commented on HBASE-5262:
------------------------------
@Mikhail We used have an extra column family in .META. called history. Into it we'd write 'events'. It was stripped because though a nice idea, it caused more trouble than it was worth (x-cluster deadlocking, interfering w/ shutdowns, growing at a different rate to the info family in .META.)...so yeah, should fail fast if can't put events, etc.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192253#comment-13192253 ]
stack commented on HBASE-5262:
------------------------------
@Mikhail Sounds good to me
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191682#comment-13191682 ]
Lars Hofhansl commented on HBASE-5262:
--------------------------------------
You thinking JSON or something?
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192491#comment-13192491 ]
Mikhail Bautin commented on HBASE-5262:
---------------------------------------
Some of the information needed for compaction decision making:
* Compaction start time
* For each compaction input file:
* Size
* Number of key/value pairs
* Average key/value size
* other metadata
* Compaction end time
* Compaction status (success, failure)
* The same metadata as above for the compaction output file
I am not yet sure what information we would like to collect for caching decision making—that needs more thinking.
We could also collect "region history", e.g. region open / close events:
* Region name
* Event type
* Server where the region was opened or closed
* Reason
That would allow to detect problematic regions that move from machine to machine automatically.
I agree that it would make sense to isolate information collection logic from decision making logic, so that external adaptive cluster tuning tools and/or external sources of information could be plugged in.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192500#comment-13192500 ]
Todd Lipcon commented on HBASE-5262:
------------------------------------
Also we should isolate the writing of this table to a separate thread with a bounded queue (which drops off the end if it becomes full). Otherwise we may run into issues where the logging of information may actually cause operations to fail, or cause deadlocks, etc. (eg a write to a region causes a flush which causes a write to the eventlog table on the same server, but all of the IPC threads are blocked because of the ongoing flush, so the eventlog write never completes). So, we should always assume that the information in the event log is potentially lossy, and only use it for heuristics and diagnosis, not for anything critical to operation.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191689#comment-13191689 ]
Mikhail Bautin commented on HBASE-5262:
---------------------------------------
JSON could be the encoding for the "value" part of each log entry. However, if we decide to store this type of information in HBase itself, we will need to think through the schema from the point of view of at least couple of different use cases, e.g. analyzing compaction performance, auto-tuning the compaction algorithm, maybe auto-tuning some block cache settings, etc.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for
monitoring and auto-tuning performance
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192464#comment-13192464 ]
Zhihong Yu commented on HBASE-5262:
-----------------------------------
Can we approach this from the perspective of what information is needed for compactions and caching decision making ?
We should also provide abstraction in providing such information so that there can be multiple sources of information the decision making can be based.
> Structured event log for HBase for monitoring and auto-tuning performance
> -------------------------------------------------------------------------
>
> Key: HBASE-5262
> URL: https://issues.apache.org/jira/browse/HBASE-5262
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
>
> Creating this JIRA to open a discussion about a structured (machine-readable) log that will record events such as compaction start/end times, compaction input/output files, their sizes, the same for flushes, etc. This can be stored e.g. in a new system table in HBase itself. The data from this log can then be analyzed and used to optimize compactions at run time, or otherwise auto-tune HBase configuration to reduce the number of knobs the user has to configure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira