You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/01/19 02:18:39 UTC

[jira] [Created] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Unit test to ensure compactions don't cache data on write
---------------------------------------------------------

                 Key: HBASE-5230
                 URL: https://issues.apache.org/jira/browse/HBASE-5230
             Project: HBase
          Issue Type: Test
            Reporter: Mikhail Bautin
            Assignee: Mikhail Bautin
            Priority: Minor


Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192479#comment-13192479 ] 

Mikhail Bautin commented on HBASE-5230:
---------------------------------------

Re-running failed unit tests locally.

Running org.apache.hadoop.hbase.master.TestSplitLogManager
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.748 sec
Running org.apache.hadoop.hbase.replication.TestReplication
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 128.254 sec
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 215.328 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.367 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.431 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.762 sec

Results :

Tests run: 39, Failures: 0, Errors: 0, Skipped: 
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191657#comment-13191657 ] 

Hadoop QA commented on HBASE-5230:
----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511585/Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated -145 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 84 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

     -1 core tests.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.replication.TestReplication
                  org.apache.hadoop.hbase.mapreduce.TestImportTsv
                  org.apache.hadoop.hbase.mapred.TestTableMapReduce
                  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
                  org.apache.hadoop.hbase.master.TestSplitLogManager

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/843//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/843//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/843//console

This message is automatically generated.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193307#comment-13193307 ] 

Phabricator commented on HBASE-5230:
------------------------------------

mbautin has abandoned the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".

  This has been committed to HBase trunk. Abandoning the diff since I forgot to include differential revision in the commit message.


REVISION DETAIL
  https://reviews.facebook.net/D1353

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190358#comment-13190358 ] 

Hadoop QA commented on HBASE-5230:
----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511373/Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated -145 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

     -1 core tests.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.regionserver.TestAtomicOperation

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/824//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/824//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/824//console

This message is automatically generated.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Ensure compactions do not cache-on-write data blocks

Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193566#comment-13193566 ] 

Hudson commented on HBASE-5230:
-------------------------------

Integrated in HBase-TRUNK-security #90 (See [https://builds.apache.org/job/HBase-TRUNK-security/90/])
    HBASE-5230 : ensure that compactions do not cache-on-write data blocks

mbautin : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

                
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Ensure compactions do not cache-on-write data blocks

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Issue Type: Improvement  (was: Test)
       Summary: Ensure compactions do not cache-on-write data blocks  (was: Unit test to ensure compactions don't cache data on write)
    
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Status: Patch Available  (was: Open)
    
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191488#comment-13191488 ] 

Phabricator commented on HBASE-5230:
------------------------------------

nspiegelberg has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:764-766 currently, there is no intelligence to estimate the resulting compacted filesize and cache compactions up to a max size, correct?
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:879-880 use this static function to write a toString method?
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4502 this comment should be changed

      // read the row, this should be a cache miss because we don't cache on compaction
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:254 add

     // TODO: need to change this test if we add a cache size threshold for compactions
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Assert.assertNull() is nice for clarity

REVISION DETAIL
  https://reviews.facebook.net/D1353

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191372#comment-13191372 ] 

Mikhail Bautin commented on HBASE-5230:
---------------------------------------

The above failed tests passed locally:

Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.187 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.841 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.529 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.111 sec
Running org.apache.hadoop.hbase.coprocessor.TestClassLoading
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.787 sec

Results :

Tests run: 24, Failures: 0, Errors: 0, Skipped: 0

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190291#comment-13190291 ] 

Phabricator commented on HBASE-5230:
------------------------------------

tedyu has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".

  Good job.

  Some minor comments.

  Please submit to Hadoop QA for test suite run.

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:746 Can this method be made package private ?
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:756 Please add javadoc for parameter isCompaction
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:880 Can we add the following line at the beginning of this method ?
  if (!LOG.isDebugEnabled()) return;
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:253 Would testCacheOnWriteOffDuringCompaction be a better name ?

  Alternatively javadoc can be added for this method to explain what it does.
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Nice.

REVISION DETAIL
  https://reviews.facebook.net/D1353

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191345#comment-13191345 ] 

Zhihong Yu commented on HBASE-5230:
-----------------------------------

TestAtomicOperation passed locally.
The patch should be good to go.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HBASE-5230:
-------------------------------

    Attachment: D1353.3.patch

mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA

  Rebasing on trunk changes.

REVISION DETAIL
  https://reviews.facebook.net/D1353

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch

A new patch addressing Nicolas's comments.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch

Attaching the most recent patch (rebased on trunk changes -- maybe even identical).
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HBASE-5230:
-------------------------------

    Attachment: D1353.4.patch

mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA

  Addressing Nicolas's comments. Re-running all unit tests.

REVISION DETAIL
  https://reviews.facebook.net/D1353

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191959#comment-13191959 ] 

stack commented on HBASE-5230:
------------------------------

lgtm
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191357#comment-13191357 ] 

Hadoop QA commented on HBASE-5230:
----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511528/Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated -145 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 84 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

     -1 core tests.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.coprocessor.TestClassLoading
                  org.apache.hadoop.hbase.mapreduce.TestImportTsv
                  org.apache.hadoop.hbase.mapred.TestTableMapReduce
                  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/840//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/840//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/840//console

This message is automatically generated.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Attachment: Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch

Attaching patch for Jenkins testing.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HBASE-5230:
-------------------------------

    Attachment: D1353.1.patch

mbautin requested code review of "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA

  Adding another test method to TestCacheOnWrite that generates some data, clears the cache, does a compaction through HRegion API, and verifies that no data blocks have been cached. When I wrote the unit test, it failed, so I modified compaction code to use an alternative CacheConfig instance with cache-on-write turned off for data blocks. This fixed the unit test. We should also be able to resolve HBASE-3976 when this goes in.

TEST PLAN
  Run all unit tests

REVISION DETAIL
  https://reviews.facebook.net/D1353

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/2817/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191320#comment-13191320 ] 

Mikhail Bautin commented on HBASE-5230:
---------------------------------------

@Ted: the unit test failure at https://builds.apache.org/job/PreCommit-HBASE-Build/824//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testRowMutationMultiThreads/ seems unrelated. Is this patch OK to be committed? (We can wait for another run of unit tests if necessary, I've just re-uploaded the patch.)
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Ensure compactions do not cache-on-write data blocks

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

      Resolution: Fixed
    Release Note: Committed into both trunk and 89-fb
          Status: Resolved  (was: Patch Available)
    
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191606#comment-13191606 ] 

Phabricator commented on HBASE-5230:
------------------------------------

mbautin has commented on the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".

  Responses to comments inline.

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:746 Done.
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:756 Done.
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:880 Replaced this with a method that "prints out" the metrics into a StringBuilder and returns a string.
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:253 Since this is for data blocks, I renamed this to testNotCachingDataBlocksDuringCompaction.
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:764-766 That is correct, to my best knowledge. Eventually, I think we would like to intelligently decide whether to cache-on-write a block based on whether the data in question is already in the block cache as part of uncompacted files: https://issues.apache.org/jira/browse/HBASE-5263

  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:879-880 In my debugging I have not yet come across a case when I would have found a SchemaMetrics.toString method useful. Also, adapting this static method to implement toString would be tricky, since it relies on getMetricsSnapshot() that takes a "snapshot" of _all_ metrics, not just those for a particular table/CF combination corresponding to one SchemaMetrics instance. Therefore, I would prefer to leave SchemaMetrics.toString() out for now.
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4502 Done.
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:254 Added.
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:295 Done.

REVISION DETAIL
  https://reviews.facebook.net/D1353

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Ensure compactions do not cache-on-write data blocks

Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193545#comment-13193545 ] 

Hudson commented on HBASE-5230:
-------------------------------

Integrated in HBase-TRUNK #2646 (See [https://builds.apache.org/job/HBase-TRUNK/2646/])
    HBASE-5230 : ensure that compactions do not cache-on-write data blocks

mbautin : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

                
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Ensure compactions do not cache-on-write data blocks

Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Bautin updated HBASE-5230:
----------------------------------

    Release Note:   (was: Committed into both trunk and 89-fb)
    
> Ensure compactions do not cache-on-write data blocks
> ----------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, D1353.4.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HBASE-5230:
-------------------------------

    Attachment: D1353.2.patch

mbautin updated the revision "[jira] [HBASE-5230] Extend TestCacheOnWrite to ensure we don't cache data blocks on compaction".
Reviewers: nspiegelberg, tedyu, Liyin, stack, JIRA

  Addressing Ted's comments. Also fixing TestFromClientSide to take into account that data blocks are not being cached on compactions.

REVISION DETAIL
  https://reviews.facebook.net/D1353

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java

                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write

Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191497#comment-13191497 ] 

Zhihong Yu commented on HBASE-5230:
-----------------------------------

This patch should be applied to 0.92, right ?
A patch for 0.92 would be desirable.
                
> Unit test to ensure compactions don't cache data on write
> ---------------------------------------------------------
>
>                 Key: HBASE-5230
>                 URL: https://issues.apache.org/jira/browse/HBASE-5230
>             Project: HBase
>          Issue Type: Test
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Minor
>         Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch
>
>
> Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira