[jira] Commented: (HBASE-2644) Investigate sync 'voodoo' splitting WALs

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875497#action_12875497 ] 

Todd Lipcon commented on HBASE-2644:
------------------------------------

Mind if I downgrade this to major, since we can't reproduce on 0.20, and no one plans on deploying on 0.21 in short term?
(if we want to start really supporting HDFS 0.21 we should add a Hudson build for trunk on 0.21)

> Investigate sync 'voodoo' splitting WALs
> ----------------------------------------
>
>                 Key: HBASE-2644
>                 URL: https://issues.apache.org/jira/browse/HBASE-2644
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> The sequence file sync seems to actually help for some weird reason when recovering parts of edits (odd -- it looks like it just adds a marker to the file).  Investigate.  This comes of review of hbase-2437.   Below is copied from http://review.hbase.org/r/74/
> {code}
> no point to call .sync() here, it just wastes a bunch of IO to write "sync markers" which we don't make any real use of.
> Cosmin Lehene 6 days, 23 hours ago (May 25th, 2010, 9:07 a.m.)
> sync() used to call syncFs(). It looks like HBASE-2544 changed things a bit, but it doesn't only add the SequenceFile sync marker.
> I added this after I've seen inconsistent results when running splitLog on bigger hlogs. Try copying a log from the cluster locally and run splitLog from the command line a few times without flushing it after each append. I used to get inconsistent results between runs and calling sync fixed it.
> There's this "//TODO: test the split of a large (lots of regions > 500 file). In my tests it seems without hflush"  in the TestHLogSplit. 
> We could do some testing to figure out why would log entries be lost when running locally.
> What would be a better way to flush the writer?
> Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:31 p.m.)
> This seems really voodoo.. if anything we're probably masking a real bug by doing this. Can you write a unit test which shows this problem (even if it takes 30 minutes to run, would be good to have in our arsenal)
> Cosmin Lehene 2 days, 18 hours ago (May 29th, 2010, 2:13 p.m.)
> I can't reproduce it on hdfs-0.20. I can't compile hdfs-0.21 (again) for some reason. I'll give it another try some other time. 
> Added the test. Also tried with a real 60MB log file. 
> I'm not sure if we should leave the test active.
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.