You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/01/19 02:18:39 UTC
[jira] [Created] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Unit test to ensure compactions don't cache data on write
---------------------------------------------------------
Key: HBASE-5230
URL: https://issues.apache.org/jira/browse/HBASE-5230
Project: HBase
Issue Type: Test
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192479#comment-13192479 ]
Mikhail Bautin commented on HBASE-5230:
---------------------------------------
Re-running failed unit tests locally.
Running org.apache.hadoop.hbase.master.TestSplitLogManager
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.748 sec
Running org.apache.hadoop.hbase.replication.TestReplication
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 128.254 sec
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 215.328 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.367 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.431 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.762 sec
Results :
Tests run: 39, Failures: 0, Errors: 0, Skipped:
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191657#comment-13191657 ]
Hadoop QA commented on HBASE-5230:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511585/Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -145 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 84 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/843//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/843//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/843//console
This message is automatically generated.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193307#comment-13193307 ]
Phabricator commented on HBASE-5230:
------------------------------------
mbautin has abandoned the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
This has been committed to HBase trunk. Abandoning the diff since I forgot to include differential revision in the commit message.
REVISION DETAIL
https://reviews.facebook.net/D1353
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190358#comment-13190358 ]
Hadoop QA commented on HBASE-5230:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511373/Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -145 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestAtomicOperation
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/824//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/824//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/824//console
This message is automatically generated.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Ensure compactions do not
cache-on-write data blocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193566#comment-13193566 ]
Hudson commented on HBASE-5230:
-------------------------------
Integrated in HBase-TRUNK-security #90 (See [https://builds.apache.org/job/HBase-TRUNK-security/90/])
HBASE-5230 : ensure that compactions do not cache-on-write data blocks
mbautin :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Ensure compactions do not
cache-on-write data blocks
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Issue Type: Improvement (was: Test)
Summary: Ensure compactions do not cache-on-write data blocks (was: Unit test to ensure compactions don't cache data on write)
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Status: Patch Available (was: Open)
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191488#comment-13191488 ]
Phabricator commented on HBASE-5230:
------------------------------------
nspiegelberg has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:764-766 currently, there is no intelligence to estimate the resulting compacted filesize and cache compactions up to a max size, correct?
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:879-880 use this static function to write a toString method?
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4502 this comment should be changed
// read the row, this should be a cache miss because we don't cache on compaction
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:254 add
// TODO: need to change this test if we add a cache size threshold for compactions
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Assert.assertNull() is nice for clarity
REVISION DETAIL
https://reviews.facebook.net/D1353
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191372#comment-13191372 ]
Mikhail Bautin commented on HBASE-5230:
---------------------------------------
The above failed tests passed locally:
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.187 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.841 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.529 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.111 sec
Running org.apache.hadoop.hbase.coprocessor.TestClassLoading
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.787 sec
Results :
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190291#comment-13190291 ]
Phabricator commented on HBASE-5230:
------------------------------------
tedyu has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Good job.
Some minor comments.
Please submit to Hadoop QA for test suite run.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:746 Can this method be made package private ?
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:756 Please add javadoc for parameter isCompaction
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:880 Can we add the following line at the beginning of this method ?
if (!LOG.isDebugEnabled()) return;
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:253 Would testCacheOnWriteOffDuringCompaction be a better name ?
Alternatively javadoc can be added for this method to explain what it does.
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Nice.
REVISION DETAIL
https://reviews.facebook.net/D1353
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191345#comment-13191345 ]
Zhihong Yu commented on HBASE-5230:
-----------------------------------
TestAtomicOperation passed locally.
The patch should be good to go.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5230:
-------------------------------
Attachment: D1353.3.patch
mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA
Rebasing on trunk changes.
REVISION DETAIL
https://reviews.facebook.net/D1353
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
A new patch addressing Nicolas's comments.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
Attaching the most recent patch (rebased on trunk changes -- maybe even identical).
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5230:
-------------------------------
Attachment: D1353.4.patch
mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA
Addressing Nicolas's comments. Re-running all unit tests.
REVISION DETAIL
https://reviews.facebook.net/D1353
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191959#comment-13191959 ]
stack commented on HBASE-5230:
------------------------------
lgtm
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191357#comment-13191357 ]
Hadoop QA commented on HBASE-5230:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511528/Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -145 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 84 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/840//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/840//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/840//console
This message is automatically generated.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
Attaching patch for Jenkins testing.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5230:
-------------------------------
Attachment: D1353.1.patch
mbautin requested code review of "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA
Adding another test method to TestCacheOnWrite that generates some data, clears the cache, does a compaction through HRegion API, and verifies that no data blocks have been cached. When I wrote the unit test, it failed, so I modified compaction code to use an alternative CacheConfig instance with cache-on-write turned off for data blocks. This fixed the unit test. We should also be able to resolve HBASE-3976 when this goes in.
TEST PLAN
Run all unit tests
REVISION DETAIL
https://reviews.facebook.net/D1353
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2817/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191320#comment-13191320 ]
Mikhail Bautin commented on HBASE-5230:
---------------------------------------
@Ted: the unit test failure at https://builds.apache.org/job/PreCommit-HBASE-Build/824//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testRowMutationMultiThreads/ seems unrelated. Is this patch OK to be committed? (We can wait for another run of unit tests if necessary, I've just re-uploaded the patch.)
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Ensure compactions do not
cache-on-write data blocks
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Resolution: Fixed
Release Note: Committed into both trunk and 89-fb
Status: Resolved (was: Patch Available)
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191606#comment-13191606 ]
Phabricator commented on HBASE-5230:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Responses to comments inline.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:746 Done.
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:756 Done.
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:880 Replaced this with a method that "prints out" the metrics into a StringBuilder and returns a string.
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:253 Since this is for data blocks, I renamed this to testNotCachingDataBlocksDuringCompaction.
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:764-766 That is correct, to my best knowledge. Eventually, I think we would like to intelligently decide whether to cache-on-write a block based on whether the data in question is already in the block cache as part of uncompacted files: https://issues.apache.org/jira/browse/HBASE-5263
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:879-880 In my debugging I have not yet come across a case when I would have found a SchemaMetrics.toString method useful. Also, adapting this static method to implement toString would be tricky, since it relies on getMetricsSnapshot() that takes a "snapshot" of _all_ metrics, not just those for a particular table/CF combination corresponding to one SchemaMetrics instance. Therefore, I would prefer to leave SchemaMetrics.toString() out for now.
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4502 Done.
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:254 Added.
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Done.
REVISION DETAIL
https://reviews.facebook.net/D1353
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Ensure compactions do not
cache-on-write data blocks
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193545#comment-13193545 ]
Hudson commented on HBASE-5230:
-------------------------------
Integrated in HBase-TRUNK #2646 (See [https://builds.apache.org/job/HBase-TRUNK/2646/])
HBASE-5230 : ensure that compactions do not cache-on-write data blocks
mbautin :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Ensure compactions do not
cache-on-write data blocks
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5230:
----------------------------------
Release Note: (was: Committed into both trunk and 89-fb)
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't
cache data on write
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5230:
-------------------------------
Attachment: D1353.2.patch
mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA
Addressing Ted's comments. Also fixing TestFromClientSide to take into account that data blocks are not being cached on compactions.
REVISION DETAIL
https://reviews.facebook.net/D1353
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions
don't cache data on write
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191497#comment-13191497 ]
Zhihong Yu commented on HBASE-5230:
-----------------------------------
This patch should be applied to 0.92, right ?
A patch for 0.92 would be desirable.
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
> Key: HBASE-5230
> URL: https://issues.apache.org/jira/browse/HBASE-5230
> Project: HBase
> Issue Type: Test
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira