You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Mike Drob <md...@apache.org> on 2018/03/27 04:35:34 UTC
Looking for AsyncWAL docs
Hi folks,
I've been working on some of the docs relating to the upcoming 2.0 release
and have struggled to find content around AsyncWAL. My impression is that
this is a pretty important new feature, yet there's nothing in the ref
guide about it.
Does it have a different name that I'm not familiar with?
If it's not in the ref guide, should I file a JIRA issue for somebody to
generate that content? Specific things that I'd be looking for are:
- How to enable/disable
- How does this impact data durability, MTTR, failover scenarios, etc.
- How does this impact replication
- Which configuration knobs exist and when would I want to tune them
As a last resort, I can try to dig through RNs in existing issues, but
that's been pretty hit or miss (mostly miss) for me so far too.
Thanks,
Mike
Re: Looking for AsyncWAL docs
Posted by Stack <st...@duboce.net>.
HBASE-20329 is first cut.
S
On Mon, Apr 2, 2018 at 2:11 PM, Stack <st...@duboce.net> wrote:
> On Mon, Apr 2, 2018 at 1:27 PM, Mike Drob <md...@apache.org> wrote:
>
>> Duo, can you include content from
>> https://issues.apache.org/jira/browse/HBASE-16689 when you are writing up
>> the docs as well? Not sure if that is talking about same or different
>> Async
>> Wal option, actually.
>>
>>
> I'll do the above.
>
> Just realized this morning that this will be a font of confusion going
> forward.
>
> You are the second one today to conflate ASYNC_WAL (i.e. not holding up
> client writes until their the sync of their write to the WAL has succeeded)
> and asycnfswal, our new asynchronous dfs writer used writing WALs.
>
> Let me try and distinguish the two better in the refguide.
>
> St.Ack
>
>
>
>
>> Mike
>>
>> On Wed, Mar 28, 2018 at 12:58 PM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>
>> > Got it. Let me read the doc for HFile v3.
>> >
>> > Will go out in the coming week so.. Hope it will not be too late...
>> >
>> > 2018-03-28 20:28 GMT+08:00 Sean Busbey <bu...@apache.org>:
>> >
>> > > HFile v3 is probably a good example.
>> > >
>> > > An appendix entry that has details that would be relevant if
>> debugging or
>> > > troublehsooting.
>> > >
>> > > And a note for the upgrade section about it being in place by default
>> and
>> > > how an operator would avoid that if they need to for some reason
>> (like an
>> > > esoteric Hadoop FS or just conservative ops position).
>> > >
>> > > I've seen conflicting statements already about the impact on
>> durability,
>> > > for example, and we should be proactive in setting expectations since
>> > folks
>> > > will be concerned as soon as they hear we have our own DFS client.
>> > >
>> > >
>> > > On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> > >
>> > > > Any examples which we could follow? This is not a user visible
>> feature,
>> > > so
>> > > > not sure what is the best way to mention it in the ref guide.
>> > > >
>> > > > 2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
>> > > >
>> > > > > Could y'all get some of this into the reference guide? Talks and
>> > > > > release notes are great, but I really want us to make sure
>> operators
>> > > > > have a nice place to figure out all the stuff we're landing in
>> 2.0.
>> > > > >
>> > > > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
>> > > > > > @Mike
>> > > > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in
>> > our
>> > > > > > HBaseCon 2016 may help you better understand the whole picture,
>> > > please
>> > > > > > check page 14 to 20 of this presentation
>> > > > > > <https://www.slideshare.net/HBaseCon/apache-hbase-
>> > > > > improvements-and-practices-at-xiaomi>
>> > > > > > on
>> > > > > > slideshare.
>> > > > > >
>> > > > > > Best Regards,
>> > > > > > Yu
>> > > > > >
>> > > > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <palomino219@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
>> > > > > >>
>> > > > > >> > Hi folks,
>> > > > > >> >
>> > > > > >> > I've been working on some of the docs relating to the
>> upcoming
>> > 2.0
>> > > > > >> release
>> > > > > >> > and have struggled to find content around AsyncWAL. My
>> > impression
>> > > is
>> > > > > that
>> > > > > >> > this is a pretty important new feature, yet there's nothing
>> in
>> > the
>> > > > ref
>> > > > > >> > guide about it.
>> > > > > >> >
>> > > > > >> > Does it have a different name that I'm not familiar with?
>> > > > > >> >
>> > > > > >> > If it's not in the ref guide, should I file a JIRA issue for
>> > > > somebody
>> > > > > to
>> > > > > >> > generate that content? Specific things that I'd be looking
>> for
>> > > are:
>> > > > > >> > - How to enable/disable
>> > > > > >> >
>> > > > > >> See HBASE-15536, just like the old way, config
>> hbase.wal.provider
>> > > > > >>
>> > > > > >> > - How does this impact data durability, MTTR, failover
>> > scenarios,
>> > > > etc.
>> > > > > >> >
>> > > > > >> Does not impact these things.
>> > > > > >>
>> > > > > >> > - How does this impact replication
>> > > > > >> >
>> > > > > >> Ditto.
>> > > > > >>
>> > > > > >> > - Which configuration knobs exist and when would I want to
>> tune
>> > > them
>> > > > > >> >
>> > > > > >> Usually you do not need to tune anything...
>> > > > > >> Before committing HBASE-15536 we have done a lot of performance
>> > > > > testings.
>> > > > > >> There are two configs which may effect performance, one
>> > > > > >> is hbase.wal.batch.size, and the other
>> > > > > >> is hbase.wal.async.use-shared-event-loop. But it is hard to
>> say
>> > > how to
>> > > > > >> tune
>> > > > > >> them...
>> > > > > >> And another thing is that, with AsyncFSWAL we can set a lower
>> > > timeout
>> > > > > when
>> > > > > >> writing WAL, but now it just shares the common dfs
>> configuration.
>> > > > Maybe
>> > > > > we
>> > > > > >> should file an issue for it.
>> > > > > >>
>> > > > > >> >
>> > > > > >> > As a last resort, I can try to dig through RNs in existing
>> > issues,
>> > > > but
>> > > > > >> > that's been pretty hit or miss (mostly miss) for me so far
>> too.
>> > > > > >> >
>> > > > > >> > I think at least we need to mention the reason why we
>> introduce
>> > > > > >> AsyncFSWAL
>> > > > > >> and make it default for 2.0 in our refguide.
>> > > > > >>
>> > > > > >> > Thanks,
>> > > > > >> > Mike
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Looking for AsyncWAL docs
Posted by Stack <st...@duboce.net>.
On Mon, Apr 2, 2018 at 1:27 PM, Mike Drob <md...@apache.org> wrote:
> Duo, can you include content from
> https://issues.apache.org/jira/browse/HBASE-16689 when you are writing up
> the docs as well? Not sure if that is talking about same or different Async
> Wal option, actually.
>
>
I'll do the above.
Just realized this morning that this will be a font of confusion going
forward.
You are the second one today to conflate ASYNC_WAL (i.e. not holding up
client writes until their the sync of their write to the WAL has succeeded)
and asycnfswal, our new asynchronous dfs writer used writing WALs.
Let me try and distinguish the two better in the refguide.
St.Ack
> Mike
>
> On Wed, Mar 28, 2018 at 12:58 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > Got it. Let me read the doc for HFile v3.
> >
> > Will go out in the coming week so.. Hope it will not be too late...
> >
> > 2018-03-28 20:28 GMT+08:00 Sean Busbey <bu...@apache.org>:
> >
> > > HFile v3 is probably a good example.
> > >
> > > An appendix entry that has details that would be relevant if debugging
> or
> > > troublehsooting.
> > >
> > > And a note for the upgrade section about it being in place by default
> and
> > > how an operator would avoid that if they need to for some reason (like
> an
> > > esoteric Hadoop FS or just conservative ops position).
> > >
> > > I've seen conflicting statements already about the impact on
> durability,
> > > for example, and we should be proactive in setting expectations since
> > folks
> > > will be concerned as soon as they hear we have our own DFS client.
> > >
> > >
> > > On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> > >
> > > > Any examples which we could follow? This is not a user visible
> feature,
> > > so
> > > > not sure what is the best way to mention it in the ref guide.
> > > >
> > > > 2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
> > > >
> > > > > Could y'all get some of this into the reference guide? Talks and
> > > > > release notes are great, but I really want us to make sure
> operators
> > > > > have a nice place to figure out all the stuff we're landing in 2.0.
> > > > >
> > > > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> > > > > > @Mike
> > > > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in
> > our
> > > > > > HBaseCon 2016 may help you better understand the whole picture,
> > > please
> > > > > > check page 14 to 20 of this presentation
> > > > > > <https://www.slideshare.net/HBaseCon/apache-hbase-
> > > > > improvements-and-practices-at-xiaomi>
> > > > > > on
> > > > > > slideshare.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> > > > > >>
> > > > > >> > Hi folks,
> > > > > >> >
> > > > > >> > I've been working on some of the docs relating to the upcoming
> > 2.0
> > > > > >> release
> > > > > >> > and have struggled to find content around AsyncWAL. My
> > impression
> > > is
> > > > > that
> > > > > >> > this is a pretty important new feature, yet there's nothing in
> > the
> > > > ref
> > > > > >> > guide about it.
> > > > > >> >
> > > > > >> > Does it have a different name that I'm not familiar with?
> > > > > >> >
> > > > > >> > If it's not in the ref guide, should I file a JIRA issue for
> > > > somebody
> > > > > to
> > > > > >> > generate that content? Specific things that I'd be looking for
> > > are:
> > > > > >> > - How to enable/disable
> > > > > >> >
> > > > > >> See HBASE-15536, just like the old way, config
> hbase.wal.provider
> > > > > >>
> > > > > >> > - How does this impact data durability, MTTR, failover
> > scenarios,
> > > > etc.
> > > > > >> >
> > > > > >> Does not impact these things.
> > > > > >>
> > > > > >> > - How does this impact replication
> > > > > >> >
> > > > > >> Ditto.
> > > > > >>
> > > > > >> > - Which configuration knobs exist and when would I want to
> tune
> > > them
> > > > > >> >
> > > > > >> Usually you do not need to tune anything...
> > > > > >> Before committing HBASE-15536 we have done a lot of performance
> > > > > testings.
> > > > > >> There are two configs which may effect performance, one
> > > > > >> is hbase.wal.batch.size, and the other
> > > > > >> is hbase.wal.async.use-shared-event-loop. But it is hard to say
> > > how to
> > > > > >> tune
> > > > > >> them...
> > > > > >> And another thing is that, with AsyncFSWAL we can set a lower
> > > timeout
> > > > > when
> > > > > >> writing WAL, but now it just shares the common dfs
> configuration.
> > > > Maybe
> > > > > we
> > > > > >> should file an issue for it.
> > > > > >>
> > > > > >> >
> > > > > >> > As a last resort, I can try to dig through RNs in existing
> > issues,
> > > > but
> > > > > >> > that's been pretty hit or miss (mostly miss) for me so far
> too.
> > > > > >> >
> > > > > >> > I think at least we need to mention the reason why we
> introduce
> > > > > >> AsyncFSWAL
> > > > > >> and make it default for 2.0 in our refguide.
> > > > > >>
> > > > > >> > Thanks,
> > > > > >> > Mike
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Looking for AsyncWAL docs
Posted by Mike Drob <md...@apache.org>.
Duo, can you include content from
https://issues.apache.org/jira/browse/HBASE-16689 when you are writing up
the docs as well? Not sure if that is talking about same or different Async
Wal option, actually.
Mike
On Wed, Mar 28, 2018 at 12:58 PM, 张铎(Duo Zhang) <pa...@gmail.com>
wrote:
> Got it. Let me read the doc for HFile v3.
>
> Will go out in the coming week so.. Hope it will not be too late...
>
> 2018-03-28 20:28 GMT+08:00 Sean Busbey <bu...@apache.org>:
>
> > HFile v3 is probably a good example.
> >
> > An appendix entry that has details that would be relevant if debugging or
> > troublehsooting.
> >
> > And a note for the upgrade section about it being in place by default and
> > how an operator would avoid that if they need to for some reason (like an
> > esoteric Hadoop FS or just conservative ops position).
> >
> > I've seen conflicting statements already about the impact on durability,
> > for example, and we should be proactive in setting expectations since
> folks
> > will be concerned as soon as they hear we have our own DFS client.
> >
> >
> > On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > > Any examples which we could follow? This is not a user visible feature,
> > so
> > > not sure what is the best way to mention it in the ref guide.
> > >
> > > 2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
> > >
> > > > Could y'all get some of this into the reference guide? Talks and
> > > > release notes are great, but I really want us to make sure operators
> > > > have a nice place to figure out all the stuff we're landing in 2.0.
> > > >
> > > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> > > > > @Mike
> > > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in
> our
> > > > > HBaseCon 2016 may help you better understand the whole picture,
> > please
> > > > > check page 14 to 20 of this presentation
> > > > > <https://www.slideshare.net/HBaseCon/apache-hbase-
> > > > improvements-and-practices-at-xiaomi>
> > > > > on
> > > > > slideshare.
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > > > >
> > > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> > > > >>
> > > > >> > Hi folks,
> > > > >> >
> > > > >> > I've been working on some of the docs relating to the upcoming
> 2.0
> > > > >> release
> > > > >> > and have struggled to find content around AsyncWAL. My
> impression
> > is
> > > > that
> > > > >> > this is a pretty important new feature, yet there's nothing in
> the
> > > ref
> > > > >> > guide about it.
> > > > >> >
> > > > >> > Does it have a different name that I'm not familiar with?
> > > > >> >
> > > > >> > If it's not in the ref guide, should I file a JIRA issue for
> > > somebody
> > > > to
> > > > >> > generate that content? Specific things that I'd be looking for
> > are:
> > > > >> > - How to enable/disable
> > > > >> >
> > > > >> See HBASE-15536, just like the old way, config hbase.wal.provider
> > > > >>
> > > > >> > - How does this impact data durability, MTTR, failover
> scenarios,
> > > etc.
> > > > >> >
> > > > >> Does not impact these things.
> > > > >>
> > > > >> > - How does this impact replication
> > > > >> >
> > > > >> Ditto.
> > > > >>
> > > > >> > - Which configuration knobs exist and when would I want to tune
> > them
> > > > >> >
> > > > >> Usually you do not need to tune anything...
> > > > >> Before committing HBASE-15536 we have done a lot of performance
> > > > testings.
> > > > >> There are two configs which may effect performance, one
> > > > >> is hbase.wal.batch.size, and the other
> > > > >> is hbase.wal.async.use-shared-event-loop. But it is hard to say
> > how to
> > > > >> tune
> > > > >> them...
> > > > >> And another thing is that, with AsyncFSWAL we can set a lower
> > timeout
> > > > when
> > > > >> writing WAL, but now it just shares the common dfs configuration.
> > > Maybe
> > > > we
> > > > >> should file an issue for it.
> > > > >>
> > > > >> >
> > > > >> > As a last resort, I can try to dig through RNs in existing
> issues,
> > > but
> > > > >> > that's been pretty hit or miss (mostly miss) for me so far too.
> > > > >> >
> > > > >> > I think at least we need to mention the reason why we introduce
> > > > >> AsyncFSWAL
> > > > >> and make it default for 2.0 in our refguide.
> > > > >>
> > > > >> > Thanks,
> > > > >> > Mike
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: Looking for AsyncWAL docs
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
Got it. Let me read the doc for HFile v3.
Will go out in the coming week so.. Hope it will not be too late...
2018-03-28 20:28 GMT+08:00 Sean Busbey <bu...@apache.org>:
> HFile v3 is probably a good example.
>
> An appendix entry that has details that would be relevant if debugging or
> troublehsooting.
>
> And a note for the upgrade section about it being in place by default and
> how an operator would avoid that if they need to for some reason (like an
> esoteric Hadoop FS or just conservative ops position).
>
> I've seen conflicting statements already about the impact on durability,
> for example, and we should be proactive in setting expectations since folks
> will be concerned as soon as they hear we have our own DFS client.
>
>
> On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > Any examples which we could follow? This is not a user visible feature,
> so
> > not sure what is the best way to mention it in the ref guide.
> >
> > 2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
> >
> > > Could y'all get some of this into the reference guide? Talks and
> > > release notes are great, but I really want us to make sure operators
> > > have a nice place to figure out all the stuff we're landing in 2.0.
> > >
> > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> > > > @Mike
> > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in our
> > > > HBaseCon 2016 may help you better understand the whole picture,
> please
> > > > check page 14 to 20 of this presentation
> > > > <https://www.slideshare.net/HBaseCon/apache-hbase-
> > > improvements-and-practices-at-xiaomi>
> > > > on
> > > > slideshare.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> > > >
> > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> > > >>
> > > >> > Hi folks,
> > > >> >
> > > >> > I've been working on some of the docs relating to the upcoming 2.0
> > > >> release
> > > >> > and have struggled to find content around AsyncWAL. My impression
> is
> > > that
> > > >> > this is a pretty important new feature, yet there's nothing in the
> > ref
> > > >> > guide about it.
> > > >> >
> > > >> > Does it have a different name that I'm not familiar with?
> > > >> >
> > > >> > If it's not in the ref guide, should I file a JIRA issue for
> > somebody
> > > to
> > > >> > generate that content? Specific things that I'd be looking for
> are:
> > > >> > - How to enable/disable
> > > >> >
> > > >> See HBASE-15536, just like the old way, config hbase.wal.provider
> > > >>
> > > >> > - How does this impact data durability, MTTR, failover scenarios,
> > etc.
> > > >> >
> > > >> Does not impact these things.
> > > >>
> > > >> > - How does this impact replication
> > > >> >
> > > >> Ditto.
> > > >>
> > > >> > - Which configuration knobs exist and when would I want to tune
> them
> > > >> >
> > > >> Usually you do not need to tune anything...
> > > >> Before committing HBASE-15536 we have done a lot of performance
> > > testings.
> > > >> There are two configs which may effect performance, one
> > > >> is hbase.wal.batch.size, and the other
> > > >> is hbase.wal.async.use-shared-event-loop. But it is hard to say
> how to
> > > >> tune
> > > >> them...
> > > >> And another thing is that, with AsyncFSWAL we can set a lower
> timeout
> > > when
> > > >> writing WAL, but now it just shares the common dfs configuration.
> > Maybe
> > > we
> > > >> should file an issue for it.
> > > >>
> > > >> >
> > > >> > As a last resort, I can try to dig through RNs in existing issues,
> > but
> > > >> > that's been pretty hit or miss (mostly miss) for me so far too.
> > > >> >
> > > >> > I think at least we need to mention the reason why we introduce
> > > >> AsyncFSWAL
> > > >> and make it default for 2.0 in our refguide.
> > > >>
> > > >> > Thanks,
> > > >> > Mike
> > > >> >
> > > >>
> > >
> >
>
Re: Looking for AsyncWAL docs
Posted by Sean Busbey <bu...@apache.org>.
HFile v3 is probably a good example.
An appendix entry that has details that would be relevant if debugging or
troublehsooting.
And a note for the upgrade section about it being in place by default and
how an operator would avoid that if they need to for some reason (like an
esoteric Hadoop FS or just conservative ops position).
I've seen conflicting statements already about the impact on durability,
for example, and we should be proactive in setting expectations since folks
will be concerned as soon as they hear we have our own DFS client.
On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> Any examples which we could follow? This is not a user visible feature, so
> not sure what is the best way to mention it in the ref guide.
>
> 2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
>
> > Could y'all get some of this into the reference guide? Talks and
> > release notes are great, but I really want us to make sure operators
> > have a nice place to figure out all the stuff we're landing in 2.0.
> >
> > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> > > @Mike
> > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in our
> > > HBaseCon 2016 may help you better understand the whole picture, please
> > > check page 14 to 20 of this presentation
> > > <https://www.slideshare.net/HBaseCon/apache-hbase-
> > improvements-and-practices-at-xiaomi>
> > > on
> > > slideshare.
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> > >
> > >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> > >>
> > >> > Hi folks,
> > >> >
> > >> > I've been working on some of the docs relating to the upcoming 2.0
> > >> release
> > >> > and have struggled to find content around AsyncWAL. My impression is
> > that
> > >> > this is a pretty important new feature, yet there's nothing in the
> ref
> > >> > guide about it.
> > >> >
> > >> > Does it have a different name that I'm not familiar with?
> > >> >
> > >> > If it's not in the ref guide, should I file a JIRA issue for
> somebody
> > to
> > >> > generate that content? Specific things that I'd be looking for are:
> > >> > - How to enable/disable
> > >> >
> > >> See HBASE-15536, just like the old way, config hbase.wal.provider
> > >>
> > >> > - How does this impact data durability, MTTR, failover scenarios,
> etc.
> > >> >
> > >> Does not impact these things.
> > >>
> > >> > - How does this impact replication
> > >> >
> > >> Ditto.
> > >>
> > >> > - Which configuration knobs exist and when would I want to tune them
> > >> >
> > >> Usually you do not need to tune anything...
> > >> Before committing HBASE-15536 we have done a lot of performance
> > testings.
> > >> There are two configs which may effect performance, one
> > >> is hbase.wal.batch.size, and the other
> > >> is hbase.wal.async.use-shared-event-loop. But it is hard to say how to
> > >> tune
> > >> them...
> > >> And another thing is that, with AsyncFSWAL we can set a lower timeout
> > when
> > >> writing WAL, but now it just shares the common dfs configuration.
> Maybe
> > we
> > >> should file an issue for it.
> > >>
> > >> >
> > >> > As a last resort, I can try to dig through RNs in existing issues,
> but
> > >> > that's been pretty hit or miss (mostly miss) for me so far too.
> > >> >
> > >> > I think at least we need to mention the reason why we introduce
> > >> AsyncFSWAL
> > >> and make it default for 2.0 in our refguide.
> > >>
> > >> > Thanks,
> > >> > Mike
> > >> >
> > >>
> >
>
Re: Looking for AsyncWAL docs
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
Any examples which we could follow? This is not a user visible feature, so
not sure what is the best way to mention it in the ref guide.
2018-03-27 23:47 GMT+08:00 Sean Busbey <bu...@apache.org>:
> Could y'all get some of this into the reference guide? Talks and
> release notes are great, but I really want us to make sure operators
> have a nice place to figure out all the stuff we're landing in 2.0.
>
> On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> > @Mike
> > FWIW, besides checking the JIRAs and codes, the talk Duo gave in our
> > HBaseCon 2016 may help you better understand the whole picture, please
> > check page 14 to 20 of this presentation
> > <https://www.slideshare.net/HBaseCon/apache-hbase-
> improvements-and-practices-at-xiaomi>
> > on
> > slideshare.
> >
> > Best Regards,
> > Yu
> >
> > On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> >> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> >>
> >> > Hi folks,
> >> >
> >> > I've been working on some of the docs relating to the upcoming 2.0
> >> release
> >> > and have struggled to find content around AsyncWAL. My impression is
> that
> >> > this is a pretty important new feature, yet there's nothing in the ref
> >> > guide about it.
> >> >
> >> > Does it have a different name that I'm not familiar with?
> >> >
> >> > If it's not in the ref guide, should I file a JIRA issue for somebody
> to
> >> > generate that content? Specific things that I'd be looking for are:
> >> > - How to enable/disable
> >> >
> >> See HBASE-15536, just like the old way, config hbase.wal.provider
> >>
> >> > - How does this impact data durability, MTTR, failover scenarios, etc.
> >> >
> >> Does not impact these things.
> >>
> >> > - How does this impact replication
> >> >
> >> Ditto.
> >>
> >> > - Which configuration knobs exist and when would I want to tune them
> >> >
> >> Usually you do not need to tune anything...
> >> Before committing HBASE-15536 we have done a lot of performance
> testings.
> >> There are two configs which may effect performance, one
> >> is hbase.wal.batch.size, and the other
> >> is hbase.wal.async.use-shared-event-loop. But it is hard to say how to
> >> tune
> >> them...
> >> And another thing is that, with AsyncFSWAL we can set a lower timeout
> when
> >> writing WAL, but now it just shares the common dfs configuration. Maybe
> we
> >> should file an issue for it.
> >>
> >> >
> >> > As a last resort, I can try to dig through RNs in existing issues, but
> >> > that's been pretty hit or miss (mostly miss) for me so far too.
> >> >
> >> > I think at least we need to mention the reason why we introduce
> >> AsyncFSWAL
> >> and make it default for 2.0 in our refguide.
> >>
> >> > Thanks,
> >> > Mike
> >> >
> >>
>
Re: Looking for AsyncWAL docs
Posted by Sean Busbey <bu...@apache.org>.
Could y'all get some of this into the reference guide? Talks and
release notes are great, but I really want us to make sure operators
have a nice place to figure out all the stuff we're landing in 2.0.
On Tue, Mar 27, 2018 at 10:13 AM, Yu Li <ca...@gmail.com> wrote:
> @Mike
> FWIW, besides checking the JIRAs and codes, the talk Duo gave in our
> HBaseCon 2016 may help you better understand the whole picture, please
> check page 14 to 20 of this presentation
> <https://www.slideshare.net/HBaseCon/apache-hbase-improvements-and-practices-at-xiaomi>
> on
> slideshare.
>
> Best Regards,
> Yu
>
> On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
>> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
>>
>> > Hi folks,
>> >
>> > I've been working on some of the docs relating to the upcoming 2.0
>> release
>> > and have struggled to find content around AsyncWAL. My impression is that
>> > this is a pretty important new feature, yet there's nothing in the ref
>> > guide about it.
>> >
>> > Does it have a different name that I'm not familiar with?
>> >
>> > If it's not in the ref guide, should I file a JIRA issue for somebody to
>> > generate that content? Specific things that I'd be looking for are:
>> > - How to enable/disable
>> >
>> See HBASE-15536, just like the old way, config hbase.wal.provider
>>
>> > - How does this impact data durability, MTTR, failover scenarios, etc.
>> >
>> Does not impact these things.
>>
>> > - How does this impact replication
>> >
>> Ditto.
>>
>> > - Which configuration knobs exist and when would I want to tune them
>> >
>> Usually you do not need to tune anything...
>> Before committing HBASE-15536 we have done a lot of performance testings.
>> There are two configs which may effect performance, one
>> is hbase.wal.batch.size, and the other
>> is hbase.wal.async.use-shared-event-loop. But it is hard to say how to
>> tune
>> them...
>> And another thing is that, with AsyncFSWAL we can set a lower timeout when
>> writing WAL, but now it just shares the common dfs configuration. Maybe we
>> should file an issue for it.
>>
>> >
>> > As a last resort, I can try to dig through RNs in existing issues, but
>> > that's been pretty hit or miss (mostly miss) for me so far too.
>> >
>> > I think at least we need to mention the reason why we introduce
>> AsyncFSWAL
>> and make it default for 2.0 in our refguide.
>>
>> > Thanks,
>> > Mike
>> >
>>
Re: Looking for AsyncWAL docs
Posted by Yu Li <ca...@gmail.com>.
@Mike
FWIW, besides checking the JIRAs and codes, the talk Duo gave in our
HBaseCon 2016 may help you better understand the whole picture, please
check page 14 to 20 of this presentation
<https://www.slideshare.net/HBaseCon/apache-hbase-improvements-and-practices-at-xiaomi>
on
slideshare.
Best Regards,
Yu
On 27 March 2018 at 14:26, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> 2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
>
> > Hi folks,
> >
> > I've been working on some of the docs relating to the upcoming 2.0
> release
> > and have struggled to find content around AsyncWAL. My impression is that
> > this is a pretty important new feature, yet there's nothing in the ref
> > guide about it.
> >
> > Does it have a different name that I'm not familiar with?
> >
> > If it's not in the ref guide, should I file a JIRA issue for somebody to
> > generate that content? Specific things that I'd be looking for are:
> > - How to enable/disable
> >
> See HBASE-15536, just like the old way, config hbase.wal.provider
>
> > - How does this impact data durability, MTTR, failover scenarios, etc.
> >
> Does not impact these things.
>
> > - How does this impact replication
> >
> Ditto.
>
> > - Which configuration knobs exist and when would I want to tune them
> >
> Usually you do not need to tune anything...
> Before committing HBASE-15536 we have done a lot of performance testings.
> There are two configs which may effect performance, one
> is hbase.wal.batch.size, and the other
> is hbase.wal.async.use-shared-event-loop. But it is hard to say how to
> tune
> them...
> And another thing is that, with AsyncFSWAL we can set a lower timeout when
> writing WAL, but now it just shares the common dfs configuration. Maybe we
> should file an issue for it.
>
> >
> > As a last resort, I can try to dig through RNs in existing issues, but
> > that's been pretty hit or miss (mostly miss) for me so far too.
> >
> > I think at least we need to mention the reason why we introduce
> AsyncFSWAL
> and make it default for 2.0 in our refguide.
>
> > Thanks,
> > Mike
> >
>
Re: Looking for AsyncWAL docs
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
2018-03-27 12:35 GMT+08:00 Mike Drob <md...@apache.org>:
> Hi folks,
>
> I've been working on some of the docs relating to the upcoming 2.0 release
> and have struggled to find content around AsyncWAL. My impression is that
> this is a pretty important new feature, yet there's nothing in the ref
> guide about it.
>
> Does it have a different name that I'm not familiar with?
>
> If it's not in the ref guide, should I file a JIRA issue for somebody to
> generate that content? Specific things that I'd be looking for are:
> - How to enable/disable
>
See HBASE-15536, just like the old way, config hbase.wal.provider
> - How does this impact data durability, MTTR, failover scenarios, etc.
>
Does not impact these things.
> - How does this impact replication
>
Ditto.
> - Which configuration knobs exist and when would I want to tune them
>
Usually you do not need to tune anything...
Before committing HBASE-15536 we have done a lot of performance testings.
There are two configs which may effect performance, one
is hbase.wal.batch.size, and the other
is hbase.wal.async.use-shared-event-loop. But it is hard to say how to tune
them...
And another thing is that, with AsyncFSWAL we can set a lower timeout when
writing WAL, but now it just shares the common dfs configuration. Maybe we
should file an issue for it.
>
> As a last resort, I can try to dig through RNs in existing issues, but
> that's been pretty hit or miss (mostly miss) for me so far too.
>
> I think at least we need to mention the reason why we introduce AsyncFSWAL
and make it default for 2.0 in our refguide.
> Thanks,
> Mike
>