[jira] Updated: (HBASE-2644) Investigate sync 'voodoo' splitting WALs

["Todd Lipcon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HBASE-2644:
-------------------------------

    Priority: Major  (was: Blocker)

IMO support for 0.21 is "nice to have" but if it doesnt quite work right at first I don't think it's a big deal. 0.21 itself is being billed as "might have some issues, 0.21.0 not for production uses"

> Investigate sync 'voodoo' splitting WALs
> ----------------------------------------
>
>                 Key: HBASE-2644
>                 URL: https://issues.apache.org/jira/browse/HBASE-2644
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.21.0
>
>
> The sequence file sync seems to actually help for some weird reason when recovering parts of edits (odd -- it looks like it just adds a marker to the file).  Investigate.  This comes of review of hbase-2437.   Below is copied from http://review.hbase.org/r/74/
> {code}
> no point to call .sync() here, it just wastes a bunch of IO to write "sync markers" which we don't make any real use of.
> Cosmin Lehene 6 days, 23 hours ago (May 25th, 2010, 9:07 a.m.)
> sync() used to call syncFs(). It looks like HBASE-2544 changed things a bit, but it doesn't only add the SequenceFile sync marker.
> I added this after I've seen inconsistent results when running splitLog on bigger hlogs. Try copying a log from the cluster locally and run splitLog from the command line a few times without flushing it after each append. I used to get inconsistent results between runs and calling sync fixed it.
> There's this "//TODO: test the split of a large (lots of regions > 500 file). In my tests it seems without hflush"  in the TestHLogSplit. 
> We could do some testing to figure out why would log entries be lost when running locally.
> What would be a better way to flush the writer?
> Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:31 p.m.)
> This seems really voodoo.. if anything we're probably masking a real bug by doing this. Can you write a unit test which shows this problem (even if it takes 30 minutes to run, would be good to have in our arsenal)
> Cosmin Lehene 2 days, 18 hours ago (May 29th, 2010, 2:13 p.m.)
> I can't reproduce it on hdfs-0.20. I can't compile hdfs-0.21 (again) for some reason. I'll give it another try some other time. 
> Added the test. Also tried with a real 60MB log file. 
> I'm not sure if we should leave the test active.
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.