You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2011/10/27 20:32:32 UTC
[jira] [Created] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
[89-fb] Fix per-store metrics aggregation
------------------------------------------
Key: HBASE-4686
URL: https://issues.apache.org/jira/browse/HBASE-4686
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4686:
-------------------------------
Attachment: D87.4.patch
mbautin updated the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Reviewers: Liyin, JIRA
Added another load test in a unit test with encoding turned on. That found some bugs similar to those that I observed in cluster testing, and I added a smaller test reproducing the same bugs (TestEncodedSeekers). Fixed the bugs by correctly restoring additional state when going to previous key/value (previously, only the vanilla BufferedDataBlockEncoder.SeekerState was restored but not algorithm-specific state). I also had to remove BitsetKeyDeltaEncoder for now because I could not fix its encoded seeker yet (it seems to have some more complicated bugs) but we are not planning to use that algorithm for now
Also, fixed the most recent comment by Ted and TestHFileBlock.testBlockHeapSize failure on a 32-bit JVM (thanks to Ted for pointing that one out as well).
REVISION DETAIL
https://reviews.facebook.net/D87
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/KeyValue.java
src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncoderBufferTooSmallException.java
src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
src/main/ruby/hbase/admin.rb
src/test/java/org/apache/hadoop/hbase/BROKE_TODO_FIX_TestAcidGuarantees.java
src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
src/test/java/org/apache/hadoop/hbase/io/encoding/RedundantKVGenerator.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestBufferedDataBlockEncoder.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadEncoded.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, D87.4.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4686:
----------------------------------
Attachment: HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
Second version of the patch.
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4686:
-------------------------------
Attachment: D87.3.patch
mbautin updated the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Reviewers: Liyin, JIRA
Addressing Liyin's comment.
REVISION DETAIL
https://reviews.facebook.net/D87
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-4686:
----------------------------------
Attachment: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140663#comment-13140663 ]
Phabricator commented on HBASE-4686:
------------------------------------
Liyin has requested changes to the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Thanks Mikhail for the patch.
There are some comments inline.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:386 [code style] Please add an empty line here
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:554 Why not move this comments to line 551 ?
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:704 [code style] please remove this empty line here
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1148 [code style] Please write the standard java doc format
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1157 [code style] Please remove the empty line here
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1219 [code style] Please add an empty line here
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:90 [code style] Please remove the empty line here
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:119 [code style] Please remove the empty line here
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Mikhail Bautin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin resolved HBASE-4686.
-----------------------------------
Resolution: Fixed
This has already been committed to trunk.
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, D87.4.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4686:
-------------------------------
Attachment: D87.2.patch
mbautin updated the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Reviewers: Liyin, JIRA
Fixing TestHeapSize after an unused field was removed from HRegion.
REVISION DETAIL
https://reviews.facebook.net/D87
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137517#comment-13137517 ]
jiraposter@reviews.apache.org commented on HBASE-4686:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2590/
-----------------------------------------------------------
Review request for hbase and Liyin Tang.
Summary
-------
In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster – thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
This addresses bug HBASE-4686.
https://issues.apache.org/jira/browse/HBASE-4686
Diffs
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6c821c0
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4a4436f
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 8776ffe
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 7fd8977
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java c03532a
Diff: https://reviews.apache.org/r/2590/diff
Testing
-------
Still running unit tests.
Thanks,
Mikhail
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140686#comment-13140686 ]
Phabricator commented on HBASE-4686:
------------------------------------
Liyin has commented on the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:386 Can you elaborate what the key and value in this tmpMap is in the javadoc?
It does no look very clear to me.
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145678#comment-13145678 ]
Phabricator commented on HBASE-4686:
------------------------------------
nspiegelberg has commented on the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
is there any functional problem with this code? I don't see any code guideline violations severe enough to make for a negative impact on inheriting the code. Maybe reading Code Complete would be a good way to adjust focus for code guideline severity on formatting that we can't automate away: http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137649#comment-13137649 ]
Mikhail Bautin commented on HBASE-4686:
---------------------------------------
Sorry for confusing file names. The most recent patch is:
https://issues.apache.org/jira/secure/attachment/12501169/HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4686:
-------------------------------
Attachment: D87.1.patch
mbautin requested code review of "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Reviewers: Liyin, JIRA
In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster – thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
TEST PLAN
Unit tests, dev cluster
REVISION DETAIL
https://reviews.facebook.net/D87
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137643#comment-13137643 ]
Mikhail Bautin commented on HBASE-4686:
---------------------------------------
I put another version of the patch on Differential: https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140702#comment-13140702 ]
Phabricator commented on HBASE-4686:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
OK, this is now becoming a code style discussion, which is not a bad thing once in a while :)
In general, the style guide for HBase seems to point to Oracle (former Sun)'s coding conventions: http://www.oracle.com/technetwork/java/codeconvtoc-136057.html, with the part pertaining to Javadoc at http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html. I guess it might not be a bad idea to actually read those docs some time...
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:386 Actually I a lot of places around the codebase don't have these empty lines inside javadoc comments, and I think it is not necessary here. Java formatter adds trailing whitespace to those empty lines, which is annoying. I did not find a requirement that these lines have to be there at http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html (although I did not read that entire style guide).
I will expand the documentation of tmpMap.
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:554 Will fix.
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:704 Agreed.
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1148 I think this saves space and is OK for one-line comments. This parses fine by Javadoc.
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1157 This empty line separates the bulky function header and the function body.
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1219 I will add one between 1218 and 1219.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:90 OK.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:119 This one is a matter of preference. There is one at the top, so I will keep the one at the bottom of the class definition.
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137761#comment-13137761 ]
Phabricator commented on HBASE-4686:
------------------------------------
Liyin has accepted the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
It looks good to me. Just one minor comments:
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:384 This api looks like very hard to use.
Maybe we can elaborate what's in the map a little bit:)
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics
aggregation
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178967#comment-13178967 ]
Phabricator commented on HBASE-4686:
------------------------------------
mbautin has abandoned the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics aggregation
".
Oops... Updated the wrong diff (I meant to update the delta encoding diff, sorry about that). Fortunately, this is a stale diff so closing.
REVISION DETAIL
https://reviews.facebook.net/D87
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, D87.4.patch, HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of StoreFile metrics over all stores in a region was replaced by overriding them every time. We saw these metrics drop by a factor of numRegions on a production cluster -- thanks to Kannan for noticing this! We need to fix the metrics and add a unit test to ensure regressions like this don't happen in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira