Re: [DISCUSS] Breakout discussion on storefile tracking storage solutions

["张铎 (Duo Zhang)" <pa...@gmail.com>]

I've filed HBASE-26064 for implementing a general framework to make the
store file tracking way pluggable.

The design doc is here:

https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.l9m8dadaxzee

张铎(Duo Zhang) <pa...@gmail.com> 于2021年6月9日周三 下午11:57写道：

> Filed HBASE-25988.
>
> And also posted a simple design doc there.
>
>
> https://docs.google.com/document/d/1Ao_jwa3ekXghAKXVpO7fbX_BXrTv3-RgxsneNULRXx0/edit?usp=sharing
>
> Wellington Chevreuil <we...@gmail.com> 于2021年6月9日周三
> 下午6:28写道：
>
>> I say go for it. Can you create it as a subtask under the
>> umbrella HBASE-24749 and PR to that feature branch? I had been working on
>> the tasks related to avoid "creating files in temp then rename" for
>> flushes
>> and compactions (HBASE-25391 and HBASE-25392), had managed to successfully
>> run it with the PersistedStoreEngine from HBASE-25395.
>>
>> Em ter., 8 de jun. de 2021 às 08:02, 张铎(Duo Zhang) <palomino219@gmail.com
>> >
>> escreveu:
>>
>> > Can not comment on the google doc so just reply here.
>> >
>> > Recently I read a bit of Hudi and Iceberg, they both use files on the
>> > filesystem for committing changes by default, so you can write to the
>> data
>> > directory directly without worrying about mess up the data files.
>> >
>> > If no big concerns, I could start a implement a POC by going with the
>> plain
>> > file approach.
>> >
>> > Thanks.
>> >
>> > Josh Elser <el...@apache.org> 于2021年5月27日周四 上午3:21写道：
>> >
>> > > Thanks Stack! (access given, as google probably told you already).
>> > >
>> > > Please keep me honest.
>> > >
>> > > On 5/26/21 12:29 PM, Stack wrote:
>> > > > And, what is there currently is a nice write-up....
>> > > > S
>> > > >
>> > > > On Wed, May 26, 2021 at 9:26 AM Stack <st...@duboce.net> wrote:
>> > > >
>> > > >> Can I have comment access please Josh?
>> > > >> S
>> > > >>
>> > > >> On Tue, May 25, 2021 at 8:24 PM Josh Elser <el...@apache.org>
>> wrote:
>> > > >>
>> > > >>> Hi folks,
>> > > >>>
>> > > >>> This is a follow-on for the HBASE-24749 discussion on storefile
>> > > >>> tracking, specifically focusing on where/how do we store the list
>> of
>> > > >>> files for each Store.
>> > > >>>
>> > > >>> I tried to capture my thoughts and the suggestions by Duo and
>> > > Wellington
>> > > >>> in this google doc [1].
>> > > >>>
>> > > >>> Please feel free to ask for edit permission (and send me a note if
>> > your
>> > > >>> email address isn't one that I would otherwise recognize :) ) to
>> > > >>> correct, improve, or expand on any other sections.
>> > > >>>
>> > > >>> FWIW, I was initially not super excited about a per-Store file,
>> but,
>> > > the
>> > > >>> more I think about it, the more I'm coming around to that idea. I
>> > think
>> > > >>> it will be more "exception-handling", but avoid the long-term
>> > > >>> operational burden of yet-another-important-system-table.
>> > > >>>
>> > > >>> - Josh
>> > > >>>
>> > > >>> [1]
>> > > >>>
>> > > >>>
>> > >
>> >
>> https://docs.google.com/document/d/1yzjvQvQfnT-M8ZgKdcQNedF8HssTnQR2loPkZtlJGVg/edit?usp=sharing
>> > > >>>
>> > > >>
>> > > >
>> > >
>> >
>>
>