can i use hdfs in this scenery ?

[kang hua <ka...@msn.com>]

Hi Todd Lipcon:I am developing a log structure -like filesystem on hdfs . I hope data will not be lost when machine that client on hanppen to crash, so I hope to flush operation log to hdfs as soon as possible for roll forword when back from crash. However , operation log is small , So I flush it very frequently 。 I use hflush methodfor above scenery ?  It is resonable?    and Can I use hdfsAvailable method(or others) to get file lenght that I hflush to but not close ? 
kanghuaThansk you 
 		 	   		  

Re: can i use hdfs in this scenery ?

[Kanghua151 <ka...@msn.com>]

3x。

发自我的 iPhone

在 2011-9-13，13:01，Todd Lipcon <to...@cloudera.com> 写道：

> Hi Kang Hua,
> 
> It sounds like the software you're building works just like HBase -
> HBase is essentially a log structure database. We indeed call hflush()
> on every edit (in addition to performing Group Commit for better
> throughput)
> 
> You might check out the HBase source and look particularly at the HLog class.
> 
> -Todd
> 
> 2011/9/12 kang hua <ka...@msn.com>:
>> Hi Todd Lipcon:
>> 
>> I am developing a log structure -like filesystem on hdfs . I hope data will
>> not be lost when machine that client on hanppen to crash, so I hope to flush
>> operation log to hdfs as soon as possible for roll forword when back from
>> crash. However , operation log is small , So I flush it very frequently 。
>> 
>> I use hflush methodfor above scenery ?  It is resonable?    and Can I
>> use hdfsAvailable method(or others) to get file lenght that I hflush to but
>> not close ?
>> 
>> kanghua
>> 
>> Thansk you
>> 
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera
> 

Re: can i use hdfs in this scenery ?

[Todd Lipcon <to...@cloudera.com>]

Hi Kang Hua,

It sounds like the software you're building works just like HBase -
HBase is essentially a log structure database. We indeed call hflush()
on every edit (in addition to performing Group Commit for better
throughput)

You might check out the HBase source and look particularly at the HLog class.

-Todd

2011/9/12 kang hua <ka...@msn.com>:
> Hi Todd Lipcon:
>
> I am developing a log structure -like filesystem on hdfs . I hope data will
> not be lost when machine that client on hanppen to crash, so I hope to flush
> operation log to hdfs as soon as possible for roll forword when back from
> crash. However , operation log is small , So I flush it very frequently 。
>
> I use hflush methodfor above scenery ?  It is resonable?    and Can I
> use hdfsAvailable method(or others) to get file lenght that I hflush to but
> not close ?
>
> kanghua
>
>  Thansk you
>



-- 
Todd Lipcon
Software Engineer, Cloudera