You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Amit Patel <am...@cloudera.com> on 2017/08/21 20:29:39 UTC
[DISCUSS] Merging Hybrid Logical Clocks with Master
Hi everyone,
I’d like to begin the discussion of merging the current HLC work with
master. Note that the current work is not ready to merge yet since there
are a few remaining issues (see below). For reference, HBASE-14070
<https://issues.apache.org/jira/browse/HBASE-14070> proposes using hybrid
logical clocks, which combine both logical and physical clocks to capture
not only causality relationship between events, but also physical time at
which events occur. HLC can be enabled on a per-table basis and at the
moment just the meta table has HLC enabled by default (the other tables
default to use the system clock). An initial design document created by
Enis Soztutar can be found here
<https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit#>.
The more recent status document created by Sai Teja Ranuva can be found here
<https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQkZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
Changes
The new classes that would be introduced include: a Clock interface with
three clock implementations mentioned above and enums for the clock and
timestamp type. The three clock implementations are hybrid logical, system
monotonic, and system clocks and each generate 64-bit timestamps.
Timestamps from the system monotonic and system clock represent physical
time and are directly backwards compatible.
Users can set the clock type of a table during table creation. A table’s
clock type cannot be changed after the table has been created. For tables
whose clock type is HLC, users would not be permitted to set their own
timestamps for mutations. Certain events such as region assignment,
unassignment, and recovery update the clocks upon receiving timestamps
externally.
Benefits
-
Hybrid logical clocks (and even system monotonic clocks) can solve a
variety of long standing issues related to time highlighted here
<https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
-
Future work with hybrid logical clocks would include enabling them with
user tables. Ideally we would want HLC to be used everywhere. Further work
such as global point-in-time snapshots can leverage HLC
Remaining issues
There are still a few remaining issues to close out before actually
merging, but I wanted to at least start the discussion. Currently I am
working on fixing remaining tests that are either failing or timing out in
the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as well
as doing local performance tests, and cluster tests. Some of the remaining
issues we are tracking are:
-
HBASE-18542 (Perf numbers)
<https://issues.apache.org/jira/browse/HBASE-18542>
-
HBASE-18508 (Fixing unit tests)
<https://issues.apache.org/jira/browse/HBASE-18508>
-
HBASE-18432 (Clock skew)
<https://issues.apache.org/jira/browse/HBASE-18432>
Some brainstorming/discussions we also are tracking:
-
(HBASE-18643) Adding transaction id to Result
<https://issues.apache.org/jira/browse/HBASE-18643>
-
(HBASE-18642) Deprecate setting of timestamp in client for HLC
<https://issues.apache.org/jira/browse/HBASE-18642>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Amit Patel <am...@cloudera.com>.
Created a ticket: Improve HLC implementation performance (
https://issues.apache.org/jira/browse/HBASE-18686).
On Fri, Aug 25, 2017 at 11:53 AM, Enis Söztutar <en...@gmail.com> wrote:
> Indeed it seems like a perf improvement in the code HLC clock is needed
> before the merge. Do you mind creating a sub ticket for this.
>
> Enis
>
> On Fri, Aug 25, 2017 at 10:54 AM, Amit Patel <am...@cloudera.com>
> wrote:
>
> > Just as an update with regard to performance: getting the time from or
> > updating the hybrid logical clock is quite expensive (see clocks
> > microbenchmark
> > <https://issues.apache.org/jira/browse/HBASE-18542?
> > focusedCommentId=16141932&page=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
> > to the extent that it would be a blocker for this work. There is however
> > certainly room for improvement given that the current implementation uses
> > synchronized blocks for both Clock#now and Clock#update.
> >
> > On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Thanks Amit for continuing on the work.
> > >
> > > I am biased, but of course would like to see this merged back to master
> > > (and possibly make it to 2.0) once the issues are ironed out. We can
> > > eliminate a very large class of problems with a mathematically sound
> > > strategy.
> > >
> > > Enis
> > >
> > > On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > Look at the pretty HLC timestamps in meta after a run:
> > > >
> > > > hbase:meta
> > > > column=table:state, timestamp=1576463246879096833,
> > > value=\x08\x00
> > > > hbase:namespace
> > > > column=table:state, timestamp=1576460369910562817,
> > > > value=\x08\x00
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:regioninfo, timestamp=1576462593304821761,
> > > > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > > > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> > > > STARTKEY => '', ENDKEY => ''}
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:seqnumDuringOpen,
> timestamp=1576462593304821761,
> > > > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:server, timestamp=1576462593304821761, value=
> > > > ve0536.halxg.cloudera.com:16020
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:serverstartcode, timestamp=1576462593304821761,
> > > > value=1503431013602
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:sn, timestamp=1576462593050017795, value=
> > > > ve0536.halxg.cloudera.com,16020,1503431013602
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:state, timestamp=1576462593304821761,
> value=OPEN
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > Nice work Amit. Idea of merging sounds good to me. Our metadata
> > system
> > > > > will run w/ a new rigor as HLC allows us leave behind a whole class
> > of
> > > > > issues.
> > > > >
> > > > > I ran the HLC branch on small cluster for a while w/ chaos. The
> ITBLL
> > > job
> > > > > passed so basically works I'd say. +1 from me on merge.
> > > > >
> > > > > St.Ack
> > > > >
> > > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <
> amit.patel@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I’d like to begin the discussion of merging the current HLC work
> > with
> > > > >> master. Note that the current work is not ready to merge yet since
> > > there
> > > > >> are a few remaining issues (see below). For reference, HBASE-14070
> > > > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes
> using
> > > > hybrid
> > > > >> logical clocks, which combine both logical and physical clocks to
> > > > capture
> > > > >> not only causality relationship between events, but also physical
> > time
> > > > at
> > > > >> which events occur. HLC can be enabled on a per-table basis and at
> > the
> > > > >> moment just the meta table has HLC enabled by default (the other
> > > tables
> > > > >> default to use the system clock). An initial design document
> created
> > > by
> > > > >> Enis Soztutar can be found here
> > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > > > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > > > >> The more recent status document created by Sai Teja Ranuva can be
> > > found
> > > > >> here
> > > > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > > > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > > > >>
> > > > >>
> > > > >> Changes
> > > > >>
> > > > >> The new classes that would be introduced include: a Clock
> interface
> > > with
> > > > >> three clock implementations mentioned above and enums for the
> clock
> > > and
> > > > >> timestamp type. The three clock implementations are hybrid
> logical,
> > > > >> system
> > > > >> monotonic, and system clocks and each generate 64-bit timestamps.
> > > > >> Timestamps from the system monotonic and system clock represent
> > > physical
> > > > >> time and are directly backwards compatible.
> > > > >>
> > > > >> Users can set the clock type of a table during table creation. A
> > > table’s
> > > > >> clock type cannot be changed after the table has been created. For
> > > > tables
> > > > >> whose clock type is HLC, users would not be permitted to set their
> > own
> > > > >> timestamps for mutations. Certain events such as region
> assignment,
> > > > >> unassignment, and recovery update the clocks upon receiving
> > timestamps
> > > > >> externally.
> > > > >>
> > > > >> Benefits
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Hybrid logical clocks (and even system monotonic clocks) can
> > solve
> > > a
> > > > >> variety of long standing issues related to time highlighted
> here
> > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> > > > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Future work with hybrid logical clocks would include enabling
> > them
> > > > with
> > > > >> user tables. Ideally we would want HLC to be used everywhere.
> > > Further
> > > > >> work
> > > > >> such as global point-in-time snapshots can leverage HLC
> > > > >>
> > > > >>
> > > > >> Remaining issues
> > > > >>
> > > > >> There are still a few remaining issues to close out before
> actually
> > > > >> merging, but I wanted to at least start the discussion. Currently
> I
> > am
> > > > >> working on fixing remaining tests that are either failing or
> timing
> > > out
> > > > in
> > > > >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/
> >
> > as
> > > > >> well
> > > > >> as doing local performance tests, and cluster tests. Some of the
> > > > remaining
> > > > >> issues we are tracking are:
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> HBASE-18542 (Perf numbers)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > > > >> -
> > > > >>
> > > > >> HBASE-18508 (Fixing unit tests)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > > > >> -
> > > > >>
> > > > >> HBASE-18432 (Clock skew)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > > > >>
> > > > >> Some brainstorming/discussions we also are tracking:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> (HBASE-18643) Adding transaction id to Result
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > > > >> -
> > > > >>
> > > > >> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Apekshit Sharma <ap...@cloudera.com>.
bq. ...but if we are recommending HLC for user tables, we have to also do
end-to-end testing to demonstrate that perf is not affected by HLC.
Agree. I think the only way of doing it right now is running ycsb
with/without HLC.
If it highlights an issue, it means there is one (there can't be false
-ves). But if it doesn't, it doesn't mean there isn't one (there can be
false +ve).
But i think it's a reasonable way of testing.
Any other ideas?
On Fri, Aug 25, 2017 at 2:36 PM, Enis Söztutar <en...@gmail.com> wrote:
> HLC will never be able to match the throughput for the system clock of
> course. I was expecting the perf to be in the 2-3x slower range, not more
> than an order of magnitude difference. The current perf is definitely
> acceptable for meta updates, but if we are recommending HLC for user
> tables, we have to also do end-to-end testing to demonstrate that perf is
> not affected by HLC.
>
> Enis
>
> On Fri, Aug 25, 2017 at 2:00 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
> > It's true that hlc clock is very slower than system, but is it really a
> > bottleneck? I mean 7mil ops/sec with 16 threads doesn't sound bottleneck
> > because:
> > - It's like 70 times of any RS' throughput (assuming 100k write
> > throughput)
> > - It's certainly enough for meta operations.
> > What say?
> >
> > But i agree that we should try to make it better and file a jira for
> > followup.
> >
> >
> > On Fri, Aug 25, 2017 at 11:53 AM, Enis Söztutar <en...@gmail.com>
> > wrote:
> >
> > > Indeed it seems like a perf improvement in the code HLC clock is needed
> > > before the merge. Do you mind creating a sub ticket for this.
> > >
> > > Enis
> > >
> > > On Fri, Aug 25, 2017 at 10:54 AM, Amit Patel <am...@cloudera.com>
> > > wrote:
> > >
> > > > Just as an update with regard to performance: getting the time from
> or
> > > > updating the hybrid logical clock is quite expensive (see clocks
> > > > microbenchmark
> > > > <https://issues.apache.org/jira/browse/HBASE-18542?
> > > > focusedCommentId=16141932&page=com.atlassian.jira.
> > > > plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
> > > > to the extent that it would be a blocker for this work. There is
> > however
> > > > certainly room for improvement given that the current implementation
> > uses
> > > > synchronized blocks for both Clock#now and Clock#update.
> > > >
> > > > On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org>
> > wrote:
> > > >
> > > > > Thanks Amit for continuing on the work.
> > > > >
> > > > > I am biased, but of course would like to see this merged back to
> > master
> > > > > (and possibly make it to 2.0) once the issues are ironed out. We
> can
> > > > > eliminate a very large class of problems with a mathematically
> sound
> > > > > strategy.
> > > > >
> > > > > Enis
> > > > >
> > > > > On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > Look at the pretty HLC timestamps in meta after a run:
> > > > > >
> > > > > > hbase:meta
> > > > > > column=table:state, timestamp=1576463246879096833,
> > > > > value=\x08\x00
> > > > > > hbase:namespace
> > > > > > column=table:state, timestamp=1576460369910562817,
> > > > > > value=\x08\x00
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:regioninfo, timestamp=1576462593304821761,
> > > > > > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > > > > > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> > b5.',
> > > > > > STARTKEY => '', ENDKEY => ''}
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:seqnumDuringOpen,
> > > timestamp=1576462593304821761,
> > > > > > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:server, timestamp=1576462593304821761,
> value=
> > > > > > ve0536.halxg.cloudera.com:16020
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:serverstartcode,
> > timestamp=1576462593304821761,
> > > > > > value=1503431013602
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:sn, timestamp=1576462593050017795, value=
> > > > > > ve0536.halxg.cloudera.com,16020,1503431013602
> > > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.
> > > > > > column=info:state, timestamp=1576462593304821761,
> > > value=OPEN
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> > > > > >
> > > > > > > Nice work Amit. Idea of merging sounds good to me. Our metadata
> > > > system
> > > > > > > will run w/ a new rigor as HLC allows us leave behind a whole
> > class
> > > > of
> > > > > > > issues.
> > > > > > >
> > > > > > > I ran the HLC branch on small cluster for a while w/ chaos. The
> > > ITBLL
> > > > > job
> > > > > > > passed so basically works I'd say. +1 from me on merge.
> > > > > > >
> > > > > > > St.Ack
> > > > > > >
> > > > > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <
> > > amit.patel@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi everyone,
> > > > > > >>
> > > > > > >> I’d like to begin the discussion of merging the current HLC
> work
> > > > with
> > > > > > >> master. Note that the current work is not ready to merge yet
> > since
> > > > > there
> > > > > > >> are a few remaining issues (see below). For reference,
> > HBASE-14070
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes
> > > using
> > > > > > hybrid
> > > > > > >> logical clocks, which combine both logical and physical clocks
> > to
> > > > > > capture
> > > > > > >> not only causality relationship between events, but also
> > physical
> > > > time
> > > > > > at
> > > > > > >> which events occur. HLC can be enabled on a per-table basis
> and
> > at
> > > > the
> > > > > > >> moment just the meta table has HLC enabled by default (the
> other
> > > > > tables
> > > > > > >> default to use the system clock). An initial design document
> > > created
> > > > > by
> > > > > > >> Enis Soztutar can be found here
> > > > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > > > > > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > > > > > >> The more recent status document created by Sai Teja Ranuva can
> > be
> > > > > found
> > > > > > >> here
> > > > > > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > > > > > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > > > > > >>
> > > > > > >>
> > > > > > >> Changes
> > > > > > >>
> > > > > > >> The new classes that would be introduced include: a Clock
> > > interface
> > > > > with
> > > > > > >> three clock implementations mentioned above and enums for the
> > > clock
> > > > > and
> > > > > > >> timestamp type. The three clock implementations are hybrid
> > > logical,
> > > > > > >> system
> > > > > > >> monotonic, and system clocks and each generate 64-bit
> > timestamps.
> > > > > > >> Timestamps from the system monotonic and system clock
> represent
> > > > > physical
> > > > > > >> time and are directly backwards compatible.
> > > > > > >>
> > > > > > >> Users can set the clock type of a table during table
> creation. A
> > > > > table’s
> > > > > > >> clock type cannot be changed after the table has been created.
> > For
> > > > > > tables
> > > > > > >> whose clock type is HLC, users would not be permitted to set
> > their
> > > > own
> > > > > > >> timestamps for mutations. Certain events such as region
> > > assignment,
> > > > > > >> unassignment, and recovery update the clocks upon receiving
> > > > timestamps
> > > > > > >> externally.
> > > > > > >>
> > > > > > >> Benefits
> > > > > > >>
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> Hybrid logical clocks (and even system monotonic clocks)
> can
> > > > solve
> > > > > a
> > > > > > >> variety of long standing issues related to time highlighted
> > > here
> > > > > > >> <https://docs.google.com/document/d/
> 1LL2GAodiYi0waBz5ODGL4L
> > > > > > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> Future work with hybrid logical clocks would include
> enabling
> > > > them
> > > > > > with
> > > > > > >> user tables. Ideally we would want HLC to be used
> everywhere.
> > > > > Further
> > > > > > >> work
> > > > > > >> such as global point-in-time snapshots can leverage HLC
> > > > > > >>
> > > > > > >>
> > > > > > >> Remaining issues
> > > > > > >>
> > > > > > >> There are still a few remaining issues to close out before
> > > actually
> > > > > > >> merging, but I wanted to at least start the discussion.
> > Currently
> > > I
> > > > am
> > > > > > >> working on fixing remaining tests that are either failing or
> > > timing
> > > > > out
> > > > > > in
> > > > > > >> the public branch <https://builds.apache.org/
> > job/HBASE-14070.HLC/
> > > >
> > > > as
> > > > > > >> well
> > > > > > >> as doing local performance tests, and cluster tests. Some of
> the
> > > > > > remaining
> > > > > > >> issues we are tracking are:
> > > > > > >>
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> HBASE-18542 (Perf numbers)
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > > > > > >> -
> > > > > > >>
> > > > > > >> HBASE-18508 (Fixing unit tests)
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > > > > > >> -
> > > > > > >>
> > > > > > >> HBASE-18432 (Clock skew)
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > > > > > >>
> > > > > > >> Some brainstorming/discussions we also are tracking:
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> (HBASE-18643) Adding transaction id to Result
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > > > > > >> -
> > > > > > >>
> > > > > > >> (HBASE-18642) Deprecate setting of timestamp in client for
> > HLC
> > > > > > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > -- Appy
> >
>
--
-- Appy
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Enis Söztutar <en...@gmail.com>.
HLC will never be able to match the throughput for the system clock of
course. I was expecting the perf to be in the 2-3x slower range, not more
than an order of magnitude difference. The current perf is definitely
acceptable for meta updates, but if we are recommending HLC for user
tables, we have to also do end-to-end testing to demonstrate that perf is
not affected by HLC.
Enis
On Fri, Aug 25, 2017 at 2:00 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> It's true that hlc clock is very slower than system, but is it really a
> bottleneck? I mean 7mil ops/sec with 16 threads doesn't sound bottleneck
> because:
> - It's like 70 times of any RS' throughput (assuming 100k write
> throughput)
> - It's certainly enough for meta operations.
> What say?
>
> But i agree that we should try to make it better and file a jira for
> followup.
>
>
> On Fri, Aug 25, 2017 at 11:53 AM, Enis Söztutar <en...@gmail.com>
> wrote:
>
> > Indeed it seems like a perf improvement in the code HLC clock is needed
> > before the merge. Do you mind creating a sub ticket for this.
> >
> > Enis
> >
> > On Fri, Aug 25, 2017 at 10:54 AM, Amit Patel <am...@cloudera.com>
> > wrote:
> >
> > > Just as an update with regard to performance: getting the time from or
> > > updating the hybrid logical clock is quite expensive (see clocks
> > > microbenchmark
> > > <https://issues.apache.org/jira/browse/HBASE-18542?
> > > focusedCommentId=16141932&page=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
> > > to the extent that it would be a blocker for this work. There is
> however
> > > certainly room for improvement given that the current implementation
> uses
> > > synchronized blocks for both Clock#now and Clock#update.
> > >
> > > On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org>
> wrote:
> > >
> > > > Thanks Amit for continuing on the work.
> > > >
> > > > I am biased, but of course would like to see this merged back to
> master
> > > > (and possibly make it to 2.0) once the issues are ironed out. We can
> > > > eliminate a very large class of problems with a mathematically sound
> > > > strategy.
> > > >
> > > > Enis
> > > >
> > > > On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > Look at the pretty HLC timestamps in meta after a run:
> > > > >
> > > > > hbase:meta
> > > > > column=table:state, timestamp=1576463246879096833,
> > > > value=\x08\x00
> > > > > hbase:namespace
> > > > > column=table:state, timestamp=1576460369910562817,
> > > > > value=\x08\x00
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:regioninfo, timestamp=1576462593304821761,
> > > > > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > > > > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ec
> b5.',
> > > > > STARTKEY => '', ENDKEY => ''}
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:seqnumDuringOpen,
> > timestamp=1576462593304821761,
> > > > > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:server, timestamp=1576462593304821761, value=
> > > > > ve0536.halxg.cloudera.com:16020
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:serverstartcode,
> timestamp=1576462593304821761,
> > > > > value=1503431013602
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:sn, timestamp=1576462593050017795, value=
> > > > > ve0536.halxg.cloudera.com,16020,1503431013602
> > > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > > column=info:state, timestamp=1576462593304821761,
> > value=OPEN
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > Nice work Amit. Idea of merging sounds good to me. Our metadata
> > > system
> > > > > > will run w/ a new rigor as HLC allows us leave behind a whole
> class
> > > of
> > > > > > issues.
> > > > > >
> > > > > > I ran the HLC branch on small cluster for a while w/ chaos. The
> > ITBLL
> > > > job
> > > > > > passed so basically works I'd say. +1 from me on merge.
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <
> > amit.patel@cloudera.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> I’d like to begin the discussion of merging the current HLC work
> > > with
> > > > > >> master. Note that the current work is not ready to merge yet
> since
> > > > there
> > > > > >> are a few remaining issues (see below). For reference,
> HBASE-14070
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes
> > using
> > > > > hybrid
> > > > > >> logical clocks, which combine both logical and physical clocks
> to
> > > > > capture
> > > > > >> not only causality relationship between events, but also
> physical
> > > time
> > > > > at
> > > > > >> which events occur. HLC can be enabled on a per-table basis and
> at
> > > the
> > > > > >> moment just the meta table has HLC enabled by default (the other
> > > > tables
> > > > > >> default to use the system clock). An initial design document
> > created
> > > > by
> > > > > >> Enis Soztutar can be found here
> > > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > > > > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > > > > >> The more recent status document created by Sai Teja Ranuva can
> be
> > > > found
> > > > > >> here
> > > > > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > > > > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > > > > >>
> > > > > >>
> > > > > >> Changes
> > > > > >>
> > > > > >> The new classes that would be introduced include: a Clock
> > interface
> > > > with
> > > > > >> three clock implementations mentioned above and enums for the
> > clock
> > > > and
> > > > > >> timestamp type. The three clock implementations are hybrid
> > logical,
> > > > > >> system
> > > > > >> monotonic, and system clocks and each generate 64-bit
> timestamps.
> > > > > >> Timestamps from the system monotonic and system clock represent
> > > > physical
> > > > > >> time and are directly backwards compatible.
> > > > > >>
> > > > > >> Users can set the clock type of a table during table creation. A
> > > > table’s
> > > > > >> clock type cannot be changed after the table has been created.
> For
> > > > > tables
> > > > > >> whose clock type is HLC, users would not be permitted to set
> their
> > > own
> > > > > >> timestamps for mutations. Certain events such as region
> > assignment,
> > > > > >> unassignment, and recovery update the clocks upon receiving
> > > timestamps
> > > > > >> externally.
> > > > > >>
> > > > > >> Benefits
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Hybrid logical clocks (and even system monotonic clocks) can
> > > solve
> > > > a
> > > > > >> variety of long standing issues related to time highlighted
> > here
> > > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> > > > > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Future work with hybrid logical clocks would include enabling
> > > them
> > > > > with
> > > > > >> user tables. Ideally we would want HLC to be used everywhere.
> > > > Further
> > > > > >> work
> > > > > >> such as global point-in-time snapshots can leverage HLC
> > > > > >>
> > > > > >>
> > > > > >> Remaining issues
> > > > > >>
> > > > > >> There are still a few remaining issues to close out before
> > actually
> > > > > >> merging, but I wanted to at least start the discussion.
> Currently
> > I
> > > am
> > > > > >> working on fixing remaining tests that are either failing or
> > timing
> > > > out
> > > > > in
> > > > > >> the public branch <https://builds.apache.org/
> job/HBASE-14070.HLC/
> > >
> > > as
> > > > > >> well
> > > > > >> as doing local performance tests, and cluster tests. Some of the
> > > > > remaining
> > > > > >> issues we are tracking are:
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> HBASE-18542 (Perf numbers)
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > > > > >> -
> > > > > >>
> > > > > >> HBASE-18508 (Fixing unit tests)
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > > > > >> -
> > > > > >>
> > > > > >> HBASE-18432 (Clock skew)
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > > > > >>
> > > > > >> Some brainstorming/discussions we also are tracking:
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> (HBASE-18643) Adding transaction id to Result
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > > > > >> -
> > > > > >>
> > > > > >> (HBASE-18642) Deprecate setting of timestamp in client for
> HLC
> > > > > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
>
> -- Appy
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Apekshit Sharma <ap...@cloudera.com>.
It's true that hlc clock is very slower than system, but is it really a
bottleneck? I mean 7mil ops/sec with 16 threads doesn't sound bottleneck
because:
- It's like 70 times of any RS' throughput (assuming 100k write throughput)
- It's certainly enough for meta operations.
What say?
But i agree that we should try to make it better and file a jira for
followup.
On Fri, Aug 25, 2017 at 11:53 AM, Enis Söztutar <en...@gmail.com> wrote:
> Indeed it seems like a perf improvement in the code HLC clock is needed
> before the merge. Do you mind creating a sub ticket for this.
>
> Enis
>
> On Fri, Aug 25, 2017 at 10:54 AM, Amit Patel <am...@cloudera.com>
> wrote:
>
> > Just as an update with regard to performance: getting the time from or
> > updating the hybrid logical clock is quite expensive (see clocks
> > microbenchmark
> > <https://issues.apache.org/jira/browse/HBASE-18542?
> > focusedCommentId=16141932&page=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
> > to the extent that it would be a blocker for this work. There is however
> > certainly room for improvement given that the current implementation uses
> > synchronized blocks for both Clock#now and Clock#update.
> >
> > On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Thanks Amit for continuing on the work.
> > >
> > > I am biased, but of course would like to see this merged back to master
> > > (and possibly make it to 2.0) once the issues are ironed out. We can
> > > eliminate a very large class of problems with a mathematically sound
> > > strategy.
> > >
> > > Enis
> > >
> > > On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > Look at the pretty HLC timestamps in meta after a run:
> > > >
> > > > hbase:meta
> > > > column=table:state, timestamp=1576463246879096833,
> > > value=\x08\x00
> > > > hbase:namespace
> > > > column=table:state, timestamp=1576460369910562817,
> > > > value=\x08\x00
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:regioninfo, timestamp=1576462593304821761,
> > > > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > > > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> > > > STARTKEY => '', ENDKEY => ''}
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:seqnumDuringOpen,
> timestamp=1576462593304821761,
> > > > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:server, timestamp=1576462593304821761, value=
> > > > ve0536.halxg.cloudera.com:16020
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:serverstartcode, timestamp=1576462593304821761,
> > > > value=1503431013602
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:sn, timestamp=1576462593050017795, value=
> > > > ve0536.halxg.cloudera.com,16020,1503431013602
> > > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > > column=info:state, timestamp=1576462593304821761,
> value=OPEN
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > Nice work Amit. Idea of merging sounds good to me. Our metadata
> > system
> > > > > will run w/ a new rigor as HLC allows us leave behind a whole class
> > of
> > > > > issues.
> > > > >
> > > > > I ran the HLC branch on small cluster for a while w/ chaos. The
> ITBLL
> > > job
> > > > > passed so basically works I'd say. +1 from me on merge.
> > > > >
> > > > > St.Ack
> > > > >
> > > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <
> amit.patel@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I’d like to begin the discussion of merging the current HLC work
> > with
> > > > >> master. Note that the current work is not ready to merge yet since
> > > there
> > > > >> are a few remaining issues (see below). For reference, HBASE-14070
> > > > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes
> using
> > > > hybrid
> > > > >> logical clocks, which combine both logical and physical clocks to
> > > > capture
> > > > >> not only causality relationship between events, but also physical
> > time
> > > > at
> > > > >> which events occur. HLC can be enabled on a per-table basis and at
> > the
> > > > >> moment just the meta table has HLC enabled by default (the other
> > > tables
> > > > >> default to use the system clock). An initial design document
> created
> > > by
> > > > >> Enis Soztutar can be found here
> > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > > > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > > > >> The more recent status document created by Sai Teja Ranuva can be
> > > found
> > > > >> here
> > > > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > > > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > > > >>
> > > > >>
> > > > >> Changes
> > > > >>
> > > > >> The new classes that would be introduced include: a Clock
> interface
> > > with
> > > > >> three clock implementations mentioned above and enums for the
> clock
> > > and
> > > > >> timestamp type. The three clock implementations are hybrid
> logical,
> > > > >> system
> > > > >> monotonic, and system clocks and each generate 64-bit timestamps.
> > > > >> Timestamps from the system monotonic and system clock represent
> > > physical
> > > > >> time and are directly backwards compatible.
> > > > >>
> > > > >> Users can set the clock type of a table during table creation. A
> > > table’s
> > > > >> clock type cannot be changed after the table has been created. For
> > > > tables
> > > > >> whose clock type is HLC, users would not be permitted to set their
> > own
> > > > >> timestamps for mutations. Certain events such as region
> assignment,
> > > > >> unassignment, and recovery update the clocks upon receiving
> > timestamps
> > > > >> externally.
> > > > >>
> > > > >> Benefits
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Hybrid logical clocks (and even system monotonic clocks) can
> > solve
> > > a
> > > > >> variety of long standing issues related to time highlighted
> here
> > > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> > > > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Future work with hybrid logical clocks would include enabling
> > them
> > > > with
> > > > >> user tables. Ideally we would want HLC to be used everywhere.
> > > Further
> > > > >> work
> > > > >> such as global point-in-time snapshots can leverage HLC
> > > > >>
> > > > >>
> > > > >> Remaining issues
> > > > >>
> > > > >> There are still a few remaining issues to close out before
> actually
> > > > >> merging, but I wanted to at least start the discussion. Currently
> I
> > am
> > > > >> working on fixing remaining tests that are either failing or
> timing
> > > out
> > > > in
> > > > >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/
> >
> > as
> > > > >> well
> > > > >> as doing local performance tests, and cluster tests. Some of the
> > > > remaining
> > > > >> issues we are tracking are:
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> HBASE-18542 (Perf numbers)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > > > >> -
> > > > >>
> > > > >> HBASE-18508 (Fixing unit tests)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > > > >> -
> > > > >>
> > > > >> HBASE-18432 (Clock skew)
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > > > >>
> > > > >> Some brainstorming/discussions we also are tracking:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> (HBASE-18643) Adding transaction id to Result
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > > > >> -
> > > > >>
> > > > >> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> > > > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
--
-- Appy
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Enis Söztutar <en...@gmail.com>.
Indeed it seems like a perf improvement in the code HLC clock is needed
before the merge. Do you mind creating a sub ticket for this.
Enis
On Fri, Aug 25, 2017 at 10:54 AM, Amit Patel <am...@cloudera.com>
wrote:
> Just as an update with regard to performance: getting the time from or
> updating the hybrid logical clock is quite expensive (see clocks
> microbenchmark
> <https://issues.apache.org/jira/browse/HBASE-18542?
> focusedCommentId=16141932&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
> to the extent that it would be a blocker for this work. There is however
> certainly room for improvement given that the current implementation uses
> synchronized blocks for both Clock#now and Clock#update.
>
> On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Thanks Amit for continuing on the work.
> >
> > I am biased, but of course would like to see this merged back to master
> > (and possibly make it to 2.0) once the issues are ironed out. We can
> > eliminate a very large class of problems with a mathematically sound
> > strategy.
> >
> > Enis
> >
> > On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> >
> > > Look at the pretty HLC timestamps in meta after a run:
> > >
> > > hbase:meta
> > > column=table:state, timestamp=1576463246879096833,
> > value=\x08\x00
> > > hbase:namespace
> > > column=table:state, timestamp=1576460369910562817,
> > > value=\x08\x00
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:regioninfo, timestamp=1576462593304821761,
> > > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> > > STARTKEY => '', ENDKEY => ''}
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:seqnumDuringOpen, timestamp=1576462593304821761,
> > > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:server, timestamp=1576462593304821761, value=
> > > ve0536.halxg.cloudera.com:16020
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:serverstartcode, timestamp=1576462593304821761,
> > > value=1503431013602
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:sn, timestamp=1576462593050017795, value=
> > > ve0536.halxg.cloudera.com,16020,1503431013602
> > > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > > column=info:state, timestamp=1576462593304821761, value=OPEN
> > >
> > > St.Ack
> > >
> > >
> > >
> > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > Nice work Amit. Idea of merging sounds good to me. Our metadata
> system
> > > > will run w/ a new rigor as HLC allows us leave behind a whole class
> of
> > > > issues.
> > > >
> > > > I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL
> > job
> > > > passed so basically works I'd say. +1 from me on merge.
> > > >
> > > > St.Ack
> > > >
> > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <amit.patel@cloudera.com
> >
> > > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I’d like to begin the discussion of merging the current HLC work
> with
> > > >> master. Note that the current work is not ready to merge yet since
> > there
> > > >> are a few remaining issues (see below). For reference, HBASE-14070
> > > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using
> > > hybrid
> > > >> logical clocks, which combine both logical and physical clocks to
> > > capture
> > > >> not only causality relationship between events, but also physical
> time
> > > at
> > > >> which events occur. HLC can be enabled on a per-table basis and at
> the
> > > >> moment just the meta table has HLC enabled by default (the other
> > tables
> > > >> default to use the system clock). An initial design document created
> > by
> > > >> Enis Soztutar can be found here
> > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > > >> The more recent status document created by Sai Teja Ranuva can be
> > found
> > > >> here
> > > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > > >>
> > > >>
> > > >> Changes
> > > >>
> > > >> The new classes that would be introduced include: a Clock interface
> > with
> > > >> three clock implementations mentioned above and enums for the clock
> > and
> > > >> timestamp type. The three clock implementations are hybrid logical,
> > > >> system
> > > >> monotonic, and system clocks and each generate 64-bit timestamps.
> > > >> Timestamps from the system monotonic and system clock represent
> > physical
> > > >> time and are directly backwards compatible.
> > > >>
> > > >> Users can set the clock type of a table during table creation. A
> > table’s
> > > >> clock type cannot be changed after the table has been created. For
> > > tables
> > > >> whose clock type is HLC, users would not be permitted to set their
> own
> > > >> timestamps for mutations. Certain events such as region assignment,
> > > >> unassignment, and recovery update the clocks upon receiving
> timestamps
> > > >> externally.
> > > >>
> > > >> Benefits
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Hybrid logical clocks (and even system monotonic clocks) can
> solve
> > a
> > > >> variety of long standing issues related to time highlighted here
> > > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> > > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > > >>
> > > >> -
> > > >>
> > > >> Future work with hybrid logical clocks would include enabling
> them
> > > with
> > > >> user tables. Ideally we would want HLC to be used everywhere.
> > Further
> > > >> work
> > > >> such as global point-in-time snapshots can leverage HLC
> > > >>
> > > >>
> > > >> Remaining issues
> > > >>
> > > >> There are still a few remaining issues to close out before actually
> > > >> merging, but I wanted to at least start the discussion. Currently I
> am
> > > >> working on fixing remaining tests that are either failing or timing
> > out
> > > in
> > > >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/>
> as
> > > >> well
> > > >> as doing local performance tests, and cluster tests. Some of the
> > > remaining
> > > >> issues we are tracking are:
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> HBASE-18542 (Perf numbers)
> > > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > > >> -
> > > >>
> > > >> HBASE-18508 (Fixing unit tests)
> > > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > > >> -
> > > >>
> > > >> HBASE-18432 (Clock skew)
> > > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > > >>
> > > >> Some brainstorming/discussions we also are tracking:
> > > >>
> > > >> -
> > > >>
> > > >> (HBASE-18643) Adding transaction id to Result
> > > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > > >> -
> > > >>
> > > >> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> > > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Amit Patel <am...@cloudera.com>.
Just as an update with regard to performance: getting the time from or
updating the hybrid logical clock is quite expensive (see clocks
microbenchmark
<https://issues.apache.org/jira/browse/HBASE-18542?focusedCommentId=16141932&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141932>)
to the extent that it would be a blocker for this work. There is however
certainly room for improvement given that the current implementation uses
synchronized blocks for both Clock#now and Clock#update.
On Wed, Aug 23, 2017 at 6:18 PM, Enis Söztutar <en...@apache.org> wrote:
> Thanks Amit for continuing on the work.
>
> I am biased, but of course would like to see this merged back to master
> (and possibly make it to 2.0) once the issues are ironed out. We can
> eliminate a very large class of problems with a mathematically sound
> strategy.
>
> Enis
>
> On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
>
> > Look at the pretty HLC timestamps in meta after a run:
> >
> > hbase:meta
> > column=table:state, timestamp=1576463246879096833,
> value=\x08\x00
> > hbase:namespace
> > column=table:state, timestamp=1576460369910562817,
> > value=\x08\x00
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:regioninfo, timestamp=1576462593304821761,
> > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> > STARTKEY => '', ENDKEY => ''}
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:seqnumDuringOpen, timestamp=1576462593304821761,
> > value=\x00\x00\x00\x00\x00\x00\x00\x15
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:server, timestamp=1576462593304821761, value=
> > ve0536.halxg.cloudera.com:16020
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:serverstartcode, timestamp=1576462593304821761,
> > value=1503431013602
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:sn, timestamp=1576462593050017795, value=
> > ve0536.halxg.cloudera.com,16020,1503431013602
> > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> > column=info:state, timestamp=1576462593304821761, value=OPEN
> >
> > St.Ack
> >
> >
> >
> > On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> >
> > > Nice work Amit. Idea of merging sounds good to me. Our metadata system
> > > will run w/ a new rigor as HLC allows us leave behind a whole class of
> > > issues.
> > >
> > > I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL
> job
> > > passed so basically works I'd say. +1 from me on merge.
> > >
> > > St.Ack
> > >
> > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <am...@cloudera.com>
> > > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I’d like to begin the discussion of merging the current HLC work with
> > >> master. Note that the current work is not ready to merge yet since
> there
> > >> are a few remaining issues (see below). For reference, HBASE-14070
> > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using
> > hybrid
> > >> logical clocks, which combine both logical and physical clocks to
> > capture
> > >> not only causality relationship between events, but also physical time
> > at
> > >> which events occur. HLC can be enabled on a per-table basis and at the
> > >> moment just the meta table has HLC enabled by default (the other
> tables
> > >> default to use the system clock). An initial design document created
> by
> > >> Enis Soztutar can be found here
> > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> > >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> > >> The more recent status document created by Sai Teja Ranuva can be
> found
> > >> here
> > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> > >>
> > >>
> > >> Changes
> > >>
> > >> The new classes that would be introduced include: a Clock interface
> with
> > >> three clock implementations mentioned above and enums for the clock
> and
> > >> timestamp type. The three clock implementations are hybrid logical,
> > >> system
> > >> monotonic, and system clocks and each generate 64-bit timestamps.
> > >> Timestamps from the system monotonic and system clock represent
> physical
> > >> time and are directly backwards compatible.
> > >>
> > >> Users can set the clock type of a table during table creation. A
> table’s
> > >> clock type cannot be changed after the table has been created. For
> > tables
> > >> whose clock type is HLC, users would not be permitted to set their own
> > >> timestamps for mutations. Certain events such as region assignment,
> > >> unassignment, and recovery update the clocks upon receiving timestamps
> > >> externally.
> > >>
> > >> Benefits
> > >>
> > >>
> > >> -
> > >>
> > >> Hybrid logical clocks (and even system monotonic clocks) can solve
> a
> > >> variety of long standing issues related to time highlighted here
> > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> > >>
> > >> -
> > >>
> > >> Future work with hybrid logical clocks would include enabling them
> > with
> > >> user tables. Ideally we would want HLC to be used everywhere.
> Further
> > >> work
> > >> such as global point-in-time snapshots can leverage HLC
> > >>
> > >>
> > >> Remaining issues
> > >>
> > >> There are still a few remaining issues to close out before actually
> > >> merging, but I wanted to at least start the discussion. Currently I am
> > >> working on fixing remaining tests that are either failing or timing
> out
> > in
> > >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as
> > >> well
> > >> as doing local performance tests, and cluster tests. Some of the
> > remaining
> > >> issues we are tracking are:
> > >>
> > >>
> > >> -
> > >>
> > >> HBASE-18542 (Perf numbers)
> > >> <https://issues.apache.org/jira/browse/HBASE-18542>
> > >> -
> > >>
> > >> HBASE-18508 (Fixing unit tests)
> > >> <https://issues.apache.org/jira/browse/HBASE-18508>
> > >> -
> > >>
> > >> HBASE-18432 (Clock skew)
> > >> <https://issues.apache.org/jira/browse/HBASE-18432>
> > >>
> > >> Some brainstorming/discussions we also are tracking:
> > >>
> > >> -
> > >>
> > >> (HBASE-18643) Adding transaction id to Result
> > >> <https://issues.apache.org/jira/browse/HBASE-18643>
> > >> -
> > >>
> > >> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> > >> <https://issues.apache.org/jira/browse/HBASE-18642>
> > >>
> > >
> > >
> >
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Enis Söztutar <en...@apache.org>.
Thanks Amit for continuing on the work.
I am biased, but of course would like to see this merged back to master
(and possibly make it to 2.0) once the issues are ironed out. We can
eliminate a very large class of problems with a mathematically sound
strategy.
Enis
On Tue, Aug 22, 2017 at 1:28 PM, Stack <st...@duboce.net> wrote:
> Look at the pretty HLC timestamps in meta after a run:
>
> hbase:meta
> column=table:state, timestamp=1576463246879096833, value=\x08\x00
> hbase:namespace
> column=table:state, timestamp=1576460369910562817,
> value=\x08\x00
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:regioninfo, timestamp=1576462593304821761,
> value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
> 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
> STARTKEY => '', ENDKEY => ''}
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:seqnumDuringOpen, timestamp=1576462593304821761,
> value=\x00\x00\x00\x00\x00\x00\x00\x15
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:server, timestamp=1576462593304821761, value=
> ve0536.halxg.cloudera.com:16020
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:serverstartcode, timestamp=1576462593304821761,
> value=1503431013602
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:sn, timestamp=1576462593050017795, value=
> ve0536.halxg.cloudera.com,16020,1503431013602
> hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
> column=info:state, timestamp=1576462593304821761, value=OPEN
>
> St.Ack
>
>
>
> On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
>
> > Nice work Amit. Idea of merging sounds good to me. Our metadata system
> > will run w/ a new rigor as HLC allows us leave behind a whole class of
> > issues.
> >
> > I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job
> > passed so basically works I'd say. +1 from me on merge.
> >
> > St.Ack
> >
> > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <am...@cloudera.com>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> I’d like to begin the discussion of merging the current HLC work with
> >> master. Note that the current work is not ready to merge yet since there
> >> are a few remaining issues (see below). For reference, HBASE-14070
> >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using
> hybrid
> >> logical clocks, which combine both logical and physical clocks to
> capture
> >> not only causality relationship between events, but also physical time
> at
> >> which events occur. HLC can be enabled on a per-table basis and at the
> >> moment just the meta table has HLC enabled by default (the other tables
> >> default to use the system clock). An initial design document created by
> >> Enis Soztutar can be found here
> >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
> >> T4e_bXy8P9h6kWC05Bhw/edit#>.
> >> The more recent status document created by Sai Teja Ranuva can be found
> >> here
> >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
> >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
> >>
> >>
> >> Changes
> >>
> >> The new classes that would be introduced include: a Clock interface with
> >> three clock implementations mentioned above and enums for the clock and
> >> timestamp type. The three clock implementations are hybrid logical,
> >> system
> >> monotonic, and system clocks and each generate 64-bit timestamps.
> >> Timestamps from the system monotonic and system clock represent physical
> >> time and are directly backwards compatible.
> >>
> >> Users can set the clock type of a table during table creation. A table’s
> >> clock type cannot be changed after the table has been created. For
> tables
> >> whose clock type is HLC, users would not be permitted to set their own
> >> timestamps for mutations. Certain events such as region assignment,
> >> unassignment, and recovery update the clocks upon receiving timestamps
> >> externally.
> >>
> >> Benefits
> >>
> >>
> >> -
> >>
> >> Hybrid logical clocks (and even system monotonic clocks) can solve a
> >> variety of long standing issues related to time highlighted here
> >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
> >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
> >>
> >> -
> >>
> >> Future work with hybrid logical clocks would include enabling them
> with
> >> user tables. Ideally we would want HLC to be used everywhere. Further
> >> work
> >> such as global point-in-time snapshots can leverage HLC
> >>
> >>
> >> Remaining issues
> >>
> >> There are still a few remaining issues to close out before actually
> >> merging, but I wanted to at least start the discussion. Currently I am
> >> working on fixing remaining tests that are either failing or timing out
> in
> >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as
> >> well
> >> as doing local performance tests, and cluster tests. Some of the
> remaining
> >> issues we are tracking are:
> >>
> >>
> >> -
> >>
> >> HBASE-18542 (Perf numbers)
> >> <https://issues.apache.org/jira/browse/HBASE-18542>
> >> -
> >>
> >> HBASE-18508 (Fixing unit tests)
> >> <https://issues.apache.org/jira/browse/HBASE-18508>
> >> -
> >>
> >> HBASE-18432 (Clock skew)
> >> <https://issues.apache.org/jira/browse/HBASE-18432>
> >>
> >> Some brainstorming/discussions we also are tracking:
> >>
> >> -
> >>
> >> (HBASE-18643) Adding transaction id to Result
> >> <https://issues.apache.org/jira/browse/HBASE-18643>
> >> -
> >>
> >> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> >> <https://issues.apache.org/jira/browse/HBASE-18642>
> >>
> >
> >
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Stack <st...@duboce.net>.
Look at the pretty HLC timestamps in meta after a run:
hbase:meta
column=table:state, timestamp=1576463246879096833, value=\x08\x00
hbase:namespace
column=table:state, timestamp=1576460369910562817, value=\x08\x00
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:regioninfo, timestamp=1576462593304821761,
value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME =>
'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.',
STARTKEY => '', ENDKEY => ''}
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:seqnumDuringOpen, timestamp=1576462593304821761,
value=\x00\x00\x00\x00\x00\x00\x00\x15
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:server, timestamp=1576462593304821761, value=
ve0536.halxg.cloudera.com:16020
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:serverstartcode, timestamp=1576462593304821761,
value=1503431013602
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:sn, timestamp=1576462593050017795, value=
ve0536.halxg.cloudera.com,16020,1503431013602
hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.
column=info:state, timestamp=1576462593304821761, value=OPEN
St.Ack
On Tue, Aug 22, 2017 at 1:26 PM, Stack <st...@duboce.net> wrote:
> Nice work Amit. Idea of merging sounds good to me. Our metadata system
> will run w/ a new rigor as HLC allows us leave behind a whole class of
> issues.
>
> I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job
> passed so basically works I'd say. +1 from me on merge.
>
> St.Ack
>
> On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <am...@cloudera.com>
> wrote:
>
>> Hi everyone,
>>
>> I’d like to begin the discussion of merging the current HLC work with
>> master. Note that the current work is not ready to merge yet since there
>> are a few remaining issues (see below). For reference, HBASE-14070
>> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using hybrid
>> logical clocks, which combine both logical and physical clocks to capture
>> not only causality relationship between events, but also physical time at
>> which events occur. HLC can be enabled on a per-table basis and at the
>> moment just the meta table has HLC enabled by default (the other tables
>> default to use the system clock). An initial design document created by
>> Enis Soztutar can be found here
>> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD
>> T4e_bXy8P9h6kWC05Bhw/edit#>.
>> The more recent status document created by Sai Teja Ranuva can be found
>> here
>> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ
>> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
>>
>>
>> Changes
>>
>> The new classes that would be introduced include: a Clock interface with
>> three clock implementations mentioned above and enums for the clock and
>> timestamp type. The three clock implementations are hybrid logical,
>> system
>> monotonic, and system clocks and each generate 64-bit timestamps.
>> Timestamps from the system monotonic and system clock represent physical
>> time and are directly backwards compatible.
>>
>> Users can set the clock type of a table during table creation. A table’s
>> clock type cannot be changed after the table has been created. For tables
>> whose clock type is HLC, users would not be permitted to set their own
>> timestamps for mutations. Certain events such as region assignment,
>> unassignment, and recovery update the clocks upon receiving timestamps
>> externally.
>>
>> Benefits
>>
>>
>> -
>>
>> Hybrid logical clocks (and even system monotonic clocks) can solve a
>> variety of long standing issues related to time highlighted here
>> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L
>> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
>>
>> -
>>
>> Future work with hybrid logical clocks would include enabling them with
>> user tables. Ideally we would want HLC to be used everywhere. Further
>> work
>> such as global point-in-time snapshots can leverage HLC
>>
>>
>> Remaining issues
>>
>> There are still a few remaining issues to close out before actually
>> merging, but I wanted to at least start the discussion. Currently I am
>> working on fixing remaining tests that are either failing or timing out in
>> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as
>> well
>> as doing local performance tests, and cluster tests. Some of the remaining
>> issues we are tracking are:
>>
>>
>> -
>>
>> HBASE-18542 (Perf numbers)
>> <https://issues.apache.org/jira/browse/HBASE-18542>
>> -
>>
>> HBASE-18508 (Fixing unit tests)
>> <https://issues.apache.org/jira/browse/HBASE-18508>
>> -
>>
>> HBASE-18432 (Clock skew)
>> <https://issues.apache.org/jira/browse/HBASE-18432>
>>
>> Some brainstorming/discussions we also are tracking:
>>
>> -
>>
>> (HBASE-18643) Adding transaction id to Result
>> <https://issues.apache.org/jira/browse/HBASE-18643>
>> -
>>
>> (HBASE-18642) Deprecate setting of timestamp in client for HLC
>> <https://issues.apache.org/jira/browse/HBASE-18642>
>>
>
>
Re: [DISCUSS] Merging Hybrid Logical Clocks with Master
Posted by Stack <st...@duboce.net>.
Nice work Amit. Idea of merging sounds good to me. Our metadata system will
run w/ a new rigor as HLC allows us leave behind a whole class of issues.
I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job
passed so basically works I'd say. +1 from me on merge.
St.Ack
On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <am...@cloudera.com> wrote:
> Hi everyone,
>
> I’d like to begin the discussion of merging the current HLC work with
> master. Note that the current work is not ready to merge yet since there
> are a few remaining issues (see below). For reference, HBASE-14070
> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using hybrid
> logical clocks, which combine both logical and physical clocks to capture
> not only causality relationship between events, but also physical time at
> which events occur. HLC can be enabled on a per-table basis and at the
> moment just the meta table has HLC enabled by default (the other tables
> default to use the system clock). An initial design document created by
> Enis Soztutar can be found here
> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_
> bXy8P9h6kWC05Bhw/edit#>.
> The more recent status document created by Sai Teja Ranuva can be found
> here
> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_
> NuF8TbQkZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>.
>
>
> Changes
>
> The new classes that would be introduced include: a Clock interface with
> three clock implementations mentioned above and enums for the clock and
> timestamp type. The three clock implementations are hybrid logical, system
> monotonic, and system clocks and each generate 64-bit timestamps.
> Timestamps from the system monotonic and system clock represent physical
> time and are directly backwards compatible.
>
> Users can set the clock type of a table during table creation. A table’s
> clock type cannot be changed after the table has been created. For tables
> whose clock type is HLC, users would not be permitted to set their own
> timestamps for mutations. Certain events such as region assignment,
> unassignment, and recovery update the clocks upon receiving timestamps
> externally.
>
> Benefits
>
>
> -
>
> Hybrid logical clocks (and even system monotonic clocks) can solve a
> variety of long standing issues related to time highlighted here
> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_
> bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>.
>
> -
>
> Future work with hybrid logical clocks would include enabling them with
> user tables. Ideally we would want HLC to be used everywhere. Further
> work
> such as global point-in-time snapshots can leverage HLC
>
>
> Remaining issues
>
> There are still a few remaining issues to close out before actually
> merging, but I wanted to at least start the discussion. Currently I am
> working on fixing remaining tests that are either failing or timing out in
> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as well
> as doing local performance tests, and cluster tests. Some of the remaining
> issues we are tracking are:
>
>
> -
>
> HBASE-18542 (Perf numbers)
> <https://issues.apache.org/jira/browse/HBASE-18542>
> -
>
> HBASE-18508 (Fixing unit tests)
> <https://issues.apache.org/jira/browse/HBASE-18508>
> -
>
> HBASE-18432 (Clock skew)
> <https://issues.apache.org/jira/browse/HBASE-18432>
>
> Some brainstorming/discussions we also are tracking:
>
> -
>
> (HBASE-18643) Adding transaction id to Result
> <https://issues.apache.org/jira/browse/HBASE-18643>
> -
>
> (HBASE-18642) Deprecate setting of timestamp in client for HLC
> <https://issues.apache.org/jira/browse/HBASE-18642>
>