HBASE 2.0 release progress

[Stephen Jiang <sy...@gmail.com>]

Hello, fellow HBASE developers,

We are making progress towards HBASE 2.0 releases.  I am using the
following queries to search for on-going HBASE 2.0 feature work items
(project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
247!  That is a lot.  At some time in near future, we definitely need to
trim down the list; otherwise, 2.0 will never be released.

For now, Matteo and I are tracking some big projects that are on-going:

(1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
- This is a blocker to have branch-2 cut.  In the past few weeks, we made
good progress and majority of implementation is done.  The patches are
under review and testing.  Matto is drafting a document for review.

(2). HBASE-14414 Backup/Restore Phase 3
- Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
was discussed and reviewed from the community (see
http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the long
discussion); and all feedback were taken care of in the latest patch.
Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
develpment of backup/restore is blocked - the backup/restore is a key
enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
another round of vote.

(3). HBASE-15179 Offheap
- This is another important feature that I think it is MUST for 2.0.  Since
stack works closely with Intel developers, he has some insight on this
project: "Intel are betting on this. Alibaba are using the offheap read
path and interested in write path too. This work is still very much up in
the air and being worked out as we go (especially the Y! Israel inmemory
compaction component).  It is a little shakey dependent on mslab pooling,
blockcache being on by default, async wal being default, and then dependent
on lots of perf and ITBLL testing."

(4). HBASE-16952 Protobuf3
- Good news, stack got the majority of work done already.  This is a MUST
for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
left.

(5). HBASE-14070 Logical Clock
- This needs to be done before 2.0 release.  At this time, seems not making
much progress.

(6). HBASE-16833 JAVA Async Client
- A lot of progress was made by Duo and his associates.  We should be in a
good shape in JAVA client when 2.0 release.

(7). HBASE-14850 C++ Async Client
- Another project that is making good progress.  I know some customer wants
this.  This is long overdue project.  Hopefully we will make those customer
happy in 2.0 release with a good performed C++ client.

Please let me know other on-going projects that needs to be in HBASE 2.0
release (stack mentioned "logical file system tier", but I am not sure
whether it would make enough progress to have a realistic chance making
2.0).  As I said at the beginning of this email, at some point soon, we
will be more picky and trim down the unfinished features in 2.0.

Happy Weekend!
Stephen

Re: HBASE 2.0 release progress

[Enis Söztutar <en...@gmail.com>]

Great summary. The master / meta hosting has to be decided as well for 2.0.
There is a thread going on for this.

I would put Hadoop3 support in the list as well.

Enis

On Fri, Nov 11, 2016 at 6:01 PM, Ted Yu <yu...@gmail.com> wrote:

> Stephen:
> w.r.t. the JIRA filter, I added a clause:
>
> project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> issuetype != Sub-task) and *updatedDate < startOfYear()* ORDER BY issuetype
> DESC
>
> and got two pages of JIRAs.
> These JIRAs haven't been touched for about a year.
>
> Looks like they can be moved to next release if there is no active work
> going on.
>
> Cheers
>
>
>
> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
>
> > Hello, fellow HBASE developers,
> >
> > We are making progress towards HBASE 2.0 releases.  I am using the
> > following queries to search for on-going HBASE 2.0 feature work items
> > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> > 247!  That is a lot.  At some time in near future, we definitely need to
> > trim down the list; otherwise, 2.0 will never be released.
> >
> > For now, Matteo and I are tracking some big projects that are on-going:
> >
> > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> > good progress and majority of implementation is done.  The patches are
> > under review and testing.  Matto is drafting a document for review.
> >
> > (2). HBASE-14414 Backup/Restore Phase 3
> > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> > was discussed and reviewed from the community (see
> > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> > long
> > discussion); and all feedback were taken care of in the latest patch.
> > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> > develpment of backup/restore is blocked - the backup/restore is a key
> > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > another round of vote.
> >
> > (3). HBASE-15179 Offheap
> > - This is another important feature that I think it is MUST for 2.0.
> Since
> > stack works closely with Intel developers, he has some insight on this
> > project: "Intel are betting on this. Alibaba are using the offheap read
> > path and interested in write path too. This work is still very much up in
> > the air and being worked out as we go (especially the Y! Israel inmemory
> > compaction component).  It is a little shakey dependent on mslab pooling,
> > blockcache being on by default, async wal being default, and then
> dependent
> > on lots of perf and ITBLL testing."
> >
> > (4). HBASE-16952 Protobuf3
> > - Good news, stack got the majority of work done already.  This is a MUST
> > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> (findbugs)
> > left.
> >
> > (5). HBASE-14070 Logical Clock
> > - This needs to be done before 2.0 release.  At this time, seems not
> making
> > much progress.
> >
> > (6). HBASE-16833 JAVA Async Client
> > - A lot of progress was made by Duo and his associates.  We should be in
> a
> > good shape in JAVA client when 2.0 release.
> >
> > (7). HBASE-14850 C++ Async Client
> > - Another project that is making good progress.  I know some customer
> wants
> > this.  This is long overdue project.  Hopefully we will make those
> customer
> > happy in 2.0 release with a good performed C++ client.
> >
> > Please let me know other on-going projects that needs to be in HBASE 2.0
> > release (stack mentioned "logical file system tier", but I am not sure
> > whether it would make enough progress to have a realistic chance making
> > 2.0).  As I said at the beginning of this email, at some point soon, we
> > will be more picky and trim down the unfinished features in 2.0.
> >
> > Happy Weekend!
> > Stephen
> >
>

Re: HBASE 2.0 release progress

[Ted Yu <yu...@gmail.com>]

Stephen:
w.r.t. the JIRA filter, I added a clause:

project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
issuetype != Sub-task) and *updatedDate < startOfYear()* ORDER BY issuetype
DESC

and got two pages of JIRAs.
These JIRAs haven't been touched for about a year.

Looks like they can be moved to next release if there is no active work
going on.

Cheers



On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
wrote:

> Hello, fellow HBASE developers,
>
> We are making progress towards HBASE 2.0 releases.  I am using the
> following queries to search for on-going HBASE 2.0 feature work items
> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> 247!  That is a lot.  At some time in near future, we definitely need to
> trim down the list; otherwise, 2.0 will never be released.
>
> For now, Matteo and I are tracking some big projects that are on-going:
>
> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> good progress and majority of implementation is done.  The patches are
> under review and testing.  Matto is drafting a document for review.
>
> (2). HBASE-14414 Backup/Restore Phase 3
> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> was discussed and reviewed from the community (see
> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> long
> discussion); and all feedback were taken care of in the latest patch.
> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> develpment of backup/restore is blocked - the backup/restore is a key
> enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> another round of vote.
>
> (3). HBASE-15179 Offheap
> - This is another important feature that I think it is MUST for 2.0.  Since
> stack works closely with Intel developers, he has some insight on this
> project: "Intel are betting on this. Alibaba are using the offheap read
> path and interested in write path too. This work is still very much up in
> the air and being worked out as we go (especially the Y! Israel inmemory
> compaction component).  It is a little shakey dependent on mslab pooling,
> blockcache being on by default, async wal being default, and then dependent
> on lots of perf and ITBLL testing."
>
> (4). HBASE-16952 Protobuf3
> - Good news, stack got the majority of work done already.  This is a MUST
> for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
> left.
>
> (5). HBASE-14070 Logical Clock
> - This needs to be done before 2.0 release.  At this time, seems not making
> much progress.
>
> (6). HBASE-16833 JAVA Async Client
> - A lot of progress was made by Duo and his associates.  We should be in a
> good shape in JAVA client when 2.0 release.
>
> (7). HBASE-14850 C++ Async Client
> - Another project that is making good progress.  I know some customer wants
> this.  This is long overdue project.  Hopefully we will make those customer
> happy in 2.0 release with a good performed C++ client.
>
> Please let me know other on-going projects that needs to be in HBASE 2.0
> release (stack mentioned "logical file system tier", but I am not sure
> whether it would make enough progress to have a realistic chance making
> 2.0).  As I said at the beginning of this email, at some point soon, we
> will be more picky and trim down the unfinished features in 2.0.
>
> Happy Weekend!
> Stephen
>

Re: HBASE 2.0 release progress

[Andrew Purtell <an...@gmail.com>]

I suspect this feature will be considered risky but I'd be willing to try it out and give it some production-like load (and with chaos). By the time we would be iterating our platform on 2.0 we'd have the type of production loads fielded that could overpressure meta on a single server. 


> On Nov 21, 2016, at 5:59 PM, Francis Liu <to...@ymail.com.INVALID> wrote:
> 
> Late to respond to this but I'd like to get "Split Meta" HBASE-11165 into HBase 2.0 as well. We've been running with a version of it internally. I'll start putting up patches soon. 
> 
>    On Friday, November 11, 2016 5:15 PM, Stephen Jiang <sy...@gmail.com> wrote:
> 
> 
> Hello, fellow HBASE developers,
> 
> We are making progress towards HBASE 2.0 releases.  I am using the
> following queries to search for on-going HBASE 2.0 feature work items
> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> 247!  That is a lot.  At some time in near future, we definitely need to
> trim down the list; otherwise, 2.0 will never be released.
> 
> For now, Matteo and I are tracking some big projects that are on-going:
> 
> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> good progress and majority of implementation is done.  The patches are
> under review and testing.  Matto is drafting a document for review.
> 
> (2). HBASE-14414 Backup/Restore Phase 3
> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> was discussed and reviewed from the community (see
> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the long
> discussion); and all feedback were taken care of in the latest patch.
> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> develpment of backup/restore is blocked - the backup/restore is a key
> enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> another round of vote.
> 
> (3). HBASE-15179 Offheap
> - This is another important feature that I think it is MUST for 2.0.  Since
> stack works closely with Intel developers, he has some insight on this
> project: "Intel are betting on this. Alibaba are using the offheap read
> path and interested in write path too. This work is still very much up in
> the air and being worked out as we go (especially the Y! Israel inmemory
> compaction component).  It is a little shakey dependent on mslab pooling,
> blockcache being on by default, async wal being default, and then dependent
> on lots of perf and ITBLL testing."
> 
> (4). HBASE-16952 Protobuf3
> - Good news, stack got the majority of work done already.  This is a MUST
> for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
> left.
> 
> (5). HBASE-14070 Logical Clock
> - This needs to be done before 2.0 release.  At this time, seems not making
> much progress.
> 
> (6). HBASE-16833 JAVA Async Client
> - A lot of progress was made by Duo and his associates.  We should be in a
> good shape in JAVA client when 2.0 release.
> 
> (7). HBASE-14850 C++ Async Client
> - Another project that is making good progress.  I know some customer wants
> this.  This is long overdue project.  Hopefully we will make those customer
> happy in 2.0 release with a good performed C++ client.
> 
> Please let me know other on-going projects that needs to be in HBASE 2.0
> release (stack mentioned "logical file system tier", but I am not sure
> whether it would make enough progress to have a realistic chance making
> 2.0).  As I said at the beginning of this email, at some point soon, we
> will be more picky and trim down the unfinished features in 2.0.
> 
> Happy Weekend!
> Stephen
> 
> 

Re: HBASE 2.0 release progress

[Francis Liu <to...@ymail.com.INVALID>]

Late to respond to this but I'd like to get "Split Meta" HBASE-11165 into HBase 2.0 as well. We've been running with a version of it internally. I'll start putting up patches soon. 

    On Friday, November 11, 2016 5:15 PM, Stephen Jiang <sy...@gmail.com> wrote:
 

 Hello, fellow HBASE developers,

We are making progress towards HBASE 2.0 releases.  I am using the
following queries to search for on-going HBASE 2.0 feature work items
(project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
247!  That is a lot.  At some time in near future, we definitely need to
trim down the list; otherwise, 2.0 will never be released.

For now, Matteo and I are tracking some big projects that are on-going:

(1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
- This is a blocker to have branch-2 cut.  In the past few weeks, we made
good progress and majority of implementation is done.  The patches are
under review and testing.  Matto is drafting a document for review.

(2). HBASE-14414 Backup/Restore Phase 3
- Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
was discussed and reviewed from the community (see
http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the long
discussion); and all feedback were taken care of in the latest patch.
Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
develpment of backup/restore is blocked - the backup/restore is a key
enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
another round of vote.

(3). HBASE-15179 Offheap
- This is another important feature that I think it is MUST for 2.0.  Since
stack works closely with Intel developers, he has some insight on this
project: "Intel are betting on this. Alibaba are using the offheap read
path and interested in write path too. This work is still very much up in
the air and being worked out as we go (especially the Y! Israel inmemory
compaction component).  It is a little shakey dependent on mslab pooling,
blockcache being on by default, async wal being default, and then dependent
on lots of perf and ITBLL testing."

(4). HBASE-16952 Protobuf3
- Good news, stack got the majority of work done already.  This is a MUST
for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
left.

(5). HBASE-14070 Logical Clock
- This needs to be done before 2.0 release.  At this time, seems not making
much progress.

(6). HBASE-16833 JAVA Async Client
- A lot of progress was made by Duo and his associates.  We should be in a
good shape in JAVA client when 2.0 release.

(7). HBASE-14850 C++ Async Client
- Another project that is making good progress.  I know some customer wants
this.  This is long overdue project.  Hopefully we will make those customer
happy in 2.0 release with a good performed C++ client.

Please let me know other on-going projects that needs to be in HBASE 2.0
release (stack mentioned "logical file system tier", but I am not sure
whether it would make enough progress to have a realistic chance making
2.0).  As I said at the beginning of this email, at some point soon, we
will be more picky and trim down the unfinished features in 2.0.

Happy Weekend!
Stephen


   

Re: HBASE 2.0 release progress

[ramkrishna vasudevan <ra...@gmail.com>]

For the offheap write path related things,
The parent JIRA
https://issues.apache.org/jira/browse/HBASE-15179

We have raised subtasks under that. Many of them are closed and there are 3
main sub-jiras now. All the 3 are now patch available.

Subtasks include
-> Reading the requests in to a ByteBuffer pool of fixed size.
-> Making ChunkPool work with direct byte buffers
-> Sizing heuristics of onheap and offheap cells (along with memstore flush
sizing).
-> Async WAL support.


There were some POCs that we done before going with the patches against
trunk. The POC reports and design docs are available in the following links

https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMu
MaJqaMNRI86C4k5S1rQ/edit
https://docs.google.com/document/d/1bYIkwNVVyYk8bgik-C5x_yc_
2hWHyi2-9KMwEFWLfzY/edit

The POC reports suggest that we get around 24% improvement with 100 threads
when using the PE tool.

In these POCs we have also used the Y! compacting memstore and its variants
called Pipeline Memstore(which is under discussion) to prove how far their
work can help offheap because it could reduce the onheap overhead of the
cells in memstore. They can explain their current state of affairs when
they do write up here.

We are now testing with AsyncWAL to see if that could be made the default
WAL because currently in HDFS DFSOutputSTream we don't have support for
bytebuffers. We need to copy the cell contents to onheap byte[]. Currenly
as part of HBASE-15786 there is a workaround we are doing to over come that.

Sorry. I thought I had sent this mail but I just saw that it is in my
Drafts.
Regards
Ram


On Mon, Nov 14, 2016 at 5:27 AM, Stack <st...@duboce.net> wrote:

> Thanks for the writeup Stephen.
>
> See below.
>
> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
>
> > Hello, fellow HBASE developers,
> >
> > We are making progress towards HBASE 2.0 releases.  I am using the
> > following queries to search for on-going HBASE 2.0 feature work items
> > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> > 247!  That is a lot.  At some time in near future, we definitely need to
> > trim down the list; otherwise, 2.0 will never be released.
> >
> >
> Agree. Suggest we all do our own review but at a certain stage, it is up to
> the RMs to make a call on what shape of 2.0 will be.
>
>
>
> > For now, Matteo and I are tracking some big projects that are on-going:
> >
> > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> > good progress and majority of implementation is done.  The patches are
> > under review and testing.  Matto is drafting a document for review.
> >
> > (2). HBASE-14414 Backup/Restore Phase 3
> > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> > was discussed and reviewed from the community (see
> > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> > long
> > discussion); and all feedback were taken care of in the latest patch.
> > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> > develpment of backup/restore is blocked - the backup/restore is a key
> > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > another round of vote.
> >
> >
> IMO, not a blocker since it ancillary function but something we should get
> in.
>
>
>
> > (3). HBASE-15179 Offheap
> > - This is another important feature that I think it is MUST for 2.0.
> Since
> > stack works closely with Intel developers, he has some insight on this
> > project: "Intel are betting on this. Alibaba are using the offheap read
> > path and interested in write path too. This work is still very much up in
> > the air and being worked out as we go (especially the Y! Israel inmemory
> > compaction component).  It is a little shakey dependent on mslab pooling,
> > blockcache being on by default, async wal being default, and then
> dependent
> > on lots of perf and ITBLL testing."
> >
> >
> The above is from a private, hurried email that does not do the current
> state justice.
>
> Ram and Anoop are the authorities on offheaping and have been keeping up
> their state in an associated doc. The lads might be persuaded do a summary
> of current state here.
>
> Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard at
> the moment doing laste pieces and perf testing. Maybe we can a status
> dumped here in dev.
>
>
>
> > (4). HBASE-16952 Protobuf3
> > - Good news, stack got the majority of work done already.  This is a MUST
> > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> (findbugs)
> > left.
> >
> >
> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to
> this list, also needed by offheaping.
>
> Will be back after taking a look at current 2.0 list.
>
> Agree that hbase2 needs to support hadoop3.
>
> St.Ack
>
>
>
>
> > (5). HBASE-14070 Logical Clock
> > - This needs to be done before 2.0 release.  At this time, seems not
> making
> > much progress.
> >
> > (6). HBASE-16833 JAVA Async Client
> > - A lot of progress was made by Duo and his associates.  We should be in
> a
> > good shape in JAVA client when 2.0 release.
> >
> > (7). HBASE-14850 C++ Async Client
> > - Another project that is making good progress.  I know some customer
> wants
> > this.  This is long overdue project.  Hopefully we will make those
> customer
> > happy in 2.0 release with a good performed C++ client.
> >
> > Please let me know other on-going projects that needs to be in HBASE 2.0
> > release (stack mentioned "logical file system tier", but I am not sure
> > whether it would make enough progress to have a realistic chance making
> > 2.0).  As I said at the beginning of this email, at some point soon, we
> > will be more picky and trim down the unfinished features in 2.0.
> >
> > Happy Weekend!
> > Stephen
> >
>

Re: HBASE 2.0 release progress

[Ted Yu <yu...@gmail.com>]

Using this lens, backup / restore should come in 2.0 with client facing
APIs properly defined.

On Mon, Nov 14, 2016 at 2:57 PM, Enis Söztutar <en...@gmail.com> wrote:

> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
> simple test:
>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
> not make into 2.0.
>
> It is a simple test, but affectively validates whether the feature is
> client-visible and have backwards compatibility implications. If it is ok
> to bring the feature in for 2.1 or later, it means that the release should
> never block on it. Otherwise, we should at least have the API / Client
> visible parts of it in 2.0 if not the full thing.
>
> Looking at the features under discussion through this lens reveals easier
> decision points I think. A couple of examples:
>  - Java async client / C++ Client: It is independent work, can come in 2.1
> or 2.2, etc. Whenever it is completed.
>  - Offheaping : transparent ti clients, so can come in incrementally in
> 2.0, 2.1, 2.2, etc.
>
> On the other hand, 1.x releases have been going on for >1.5 years, and will
> likely go on for another year at least. So the choices (dependency, APIs,
> etc) that we make now for 2.0 will likely live for >2 years. In that
> respect I would rather be a bit more aggressive in terms of dropping
> support for older stuff and updating wherever we can. For example require
> Hadoop-2.6+, Java-8, etc.
>
> Enis
>
> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
> > Great to see progress here.
> >
> > As I'm reading through the list, here're some high-level questions I
> have:
> >
> >  -  Regarding the work on new AssignmentManager - any notes on
> > perf/stability testing? Are you guys running tips of master branch
> through
> > ITBLL setup?
> >  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
> current
> > up-to-date state of offheaping?
> >  -  Logical clock - at this point is it more like a nice to have feature
> > than "need to be done before 2.0"? What are the features blocked or
> > affected by lack thereof?
> >
> > -Mikhail
> >
> > On Sun, Nov 13, 2016 at 3:57 PM, Stack <st...@duboce.net> wrote:
> >
> > > Thanks for the writeup Stephen.
> > >
> > > See below.
> > >
> > > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
> syuanjiangdev@gmail.com>
> > > wrote:
> > >
> > > > Hello, fellow HBASE developers,
> > > >
> > > > We are making progress towards HBASE 2.0 releases.  I am using the
> > > > following queries to search for on-going HBASE 2.0 feature work items
> > > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
> > AND
> > > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
> have
> > > > 247!  That is a lot.  At some time in near future, we definitely need
> > to
> > > > trim down the list; otherwise, 2.0 will never be released.
> > > >
> > > >
> > > Agree. Suggest we all do our own review but at a certain stage, it is
> up
> > to
> > > the RMs to make a call on what shape of 2.0 will be.
> > >
> > >
> > >
> > > > For now, Matteo and I are tracking some big projects that are
> on-going:
> > > >
> > > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > > > - This is a blocker to have branch-2 cut.  In the past few weeks, we
> > made
> > > > good progress and majority of implementation is done.  The patches
> are
> > > > under review and testing.  Matto is drafting a document for review.
> > > >
> > > > (2). HBASE-14414 Backup/Restore Phase 3
> > > > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
> > patches
> > > > was discussed and reviewed from the community (see
> > > > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
> the
> > > > long
> > > > discussion); and all feedback were taken care of in the latest patch.
> > > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
> > further
> > > > develpment of backup/restore is blocked - the backup/restore is a key
> > > > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > > > another round of vote.
> > > >
> > > >
> > > IMO, not a blocker since it ancillary function but something we should
> > get
> > > in.
> > >
> > >
> > >
> > > > (3). HBASE-15179 Offheap
> > > > - This is another important feature that I think it is MUST for 2.0.
> > > Since
> > > > stack works closely with Intel developers, he has some insight on
> this
> > > > project: "Intel are betting on this. Alibaba are using the offheap
> read
> > > > path and interested in write path too. This work is still very much
> up
> > in
> > > > the air and being worked out as we go (especially the Y! Israel
> > inmemory
> > > > compaction component).  It is a little shakey dependent on mslab
> > pooling,
> > > > blockcache being on by default, async wal being default, and then
> > > dependent
> > > > on lots of perf and ITBLL testing."
> > > >
> > > >
> > > The above is from a private, hurried email that does not do the current
> > > state justice.
> > >
> > > Ram and Anoop are the authorities on offheaping and have been keeping
> up
> > > their state in an associated doc. The lads might be persuaded do a
> > summary
> > > of current state here.
> > >
> > > Ditto for inmemory compaction. Our Anastasia and Eshcar are working
> hard
> > at
> > > the moment doing laste pieces and perf testing. Maybe we can a status
> > > dumped here in dev.
> > >
> > >
> > >
> > > > (4). HBASE-16952 Protobuf3
> > > > - Good news, stack got the majority of work done already.  This is a
> > MUST
> > > > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> > > (findbugs)
> > > > left.
> > > >
> > > >
> > > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
> to
> > > this list, also needed by offheaping.
> > >
> > > Will be back after taking a look at current 2.0 list.
> > >
> > > Agree that hbase2 needs to support hadoop3.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > (5). HBASE-14070 Logical Clock
> > > > - This needs to be done before 2.0 release.  At this time, seems not
> > > making
> > > > much progress.
> > > >
> > > > (6). HBASE-16833 JAVA Async Client
> > > > - A lot of progress was made by Duo and his associates.  We should be
> > in
> > > a
> > > > good shape in JAVA client when 2.0 release.
> > > >
> > > > (7). HBASE-14850 C++ Async Client
> > > > - Another project that is making good progress.  I know some customer
> > > wants
> > > > this.  This is long overdue project.  Hopefully we will make those
> > > customer
> > > > happy in 2.0 release with a good performed C++ client.
> > > >
> > > > Please let me know other on-going projects that needs to be in HBASE
> > 2.0
> > > > release (stack mentioned "logical file system tier", but I am not
> sure
> > > > whether it would make enough progress to have a realistic chance
> making
> > > > 2.0).  As I said at the beginning of this email, at some point soon,
> we
> > > > will be more picky and trim down the unfinished features in 2.0.
> > > >
> > > > Happy Weekend!
> > > > Stephen
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
> >
>

Re: HBASE 2.0 release progress

[Andrew Purtell <an...@gmail.com>]

Favored nodes version 1 did not see much adoption and was hard to use or unusable, depending on who you ask, so please expect a focus on usability and feature completeness when/if v2 is reviewed for possible inclusion in 2.0. While I am not suggesting this is the case, if the v2 candidate shares the feature incompleteness or usability problems of its predecessor it should not go in. 


> On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan <th...@yahoo-inc.com.INVALID> wrote:
> 
> We would like to add the favored node enhancements as part of https://issues.apache.org/jira/browse/HBASE-15531 to 2.0. Most of the code should be new, there will be very less code changes to Master/AM etc.
> 
> -Thiruvel
> 
> On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John <an...@gmail.com> wrote:>-  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
> up-to-date state of offheaping?
> 
> Off heaping the read path (HBASE-11425) this is completed some time
> back.  As per my knowledge at least 2 users back ported this work to
> 1.x based version and even running in production. Am leaving it them
> for giving details..
> 
> Write path off heaping HBASE-15179 is the umbrella issue.  As u can
> see most of the sub tasks are done by now..  Mainly 2 more sub tasks
> are yet to get committed.  Both are some what bigger sized also..
> Patch is already available for them.  We hope it can be completed by
> Nov end.
> 
> Some more off heaping work, (in compaction path etc) might be taken up
> after write path work. Alibaba guys might be joining with that.  Had
> some discuss with them.. We will come up with jira and doc for that
> before actual work begin.  But as Enis said ya all that can go in 2.x
> (x>0) :-)
> 
> Just giving some high level status update. Thanks ..
> 
> -Anoop-
> 
> 
>> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>> Enis, by your criteria it seems the log4j2 upgrade ticket should be
>> considered for 2.0 as well: HBASE-12341.
>> 
>>> On Monday, November 14, 2016, Enis Söztutar <en...@gmail.com> wrote:
>>> 
>>> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
>>> simple test:
>>>   - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
>>> not make into 2.0.
>>> 
>>> It is a simple test, but affectively validates whether the feature is
>>> client-visible and have backwards compatibility implications. If it is ok
>>> to bring the feature in for 2.1 or later, it means that the release should
>>> never block on it. Otherwise, we should at least have the API / Client
>>> visible parts of it in 2.0 if not the full thing.
>>> 
>>> Looking at the features under discussion through this lens reveals easier
>>> decision points I think. A couple of examples:
>>>   - Java async client / C++ Client: It is independent work, can come in 2.1
>>> or 2.2, etc. Whenever it is completed.
>>>   - Offheaping : transparent ti clients, so can come in incrementally in
>>> 2.0, 2.1, 2.2, etc.
>>> 
>>> On the other hand, 1.x releases have been going on for >1.5 years, and will
>>> likely go on for another year at least. So the choices (dependency, APIs,
>>> etc) that we make now for 2.0 will likely live for >2 years. In that
>>> respect I would rather be a bit more aggressive in terms of dropping
>>> support for older stuff and updating wherever we can. For example require
>>> Hadoop-2.6+, Java-8, etc.
>>> 
>>> Enis
>>> 
>>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
>>> <javascript:;>>
>>> wrote:
>>> 
>>>> Great to see progress here.
>>>> 
>>>> As I'm reading through the list, here're some high-level questions I
>>> have:
>>>> 
>>>>   -  Regarding the work on new AssignmentManager - any notes on
>>>> perf/stability testing? Are you guys running tips of master branch
>>> through
>>>> ITBLL setup?
>>>>   -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
>>> current
>>>> up-to-date state of offheaping?
>>>>   -  Logical clock - at this point is it more like a nice to have feature
>>>> than "need to be done before 2.0"? What are the features blocked or
>>>> affected by lack thereof?
>>>> 
>>>> -Mikhail
>>>> 
>>>> On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
>>> wrote:
>>>> 
>>>>> Thanks for the writeup Stephen.
>>>>> 
>>>>> See below.
>>>>> 
>>>>> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
>>> syuanjiangdev@gmail.com <javascript:;>>
>>>>> wrote:
>>>>> 
>>>>>> Hello, fellow HBASE developers,
>>>>>> 
>>>>>> We are making progress towards HBASE 2.0 releases.  I am using the
>>>>>> following queries to search for on-going HBASE 2.0 feature work items
>>>>>> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
>>>> AND
>>>>>> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
>>>>>> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
>>> have
>>>>>> 247!  That is a lot.  At some time in near future, we definitely need
>>>> to
>>>>>> trim down the list; otherwise, 2.0 will never be released.
>>>>>> 
>>>>>> 
>>>>> Agree. Suggest we all do our own review but at a certain stage, it is
>>> up
>>>> to
>>>>> the RMs to make a call on what shape of 2.0 will be.
>>>>> 
>>>>> 
>>>>> 
>>>>>> For now, Matteo and I are tracking some big projects that are
>>> on-going:
>>>>>> 
>>>>>> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
>>>>>> - This is a blocker to have branch-2 cut.  In the past few weeks, we
>>>> made
>>>>>> good progress and majority of implementation is done.  The patches
>>> are
>>>>>> under review and testing.  Matto is drafting a document for review.
>>>>>> 
>>>>>> (2). HBASE-14414 Backup/Restore Phase 3
>>>>>> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
>>>> patches
>>>>>> was discussed and reviewed from the community (see
>>>>>> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
>>> the
>>>>>> long
>>>>>> discussion); and all feedback were taken care of in the latest patch.
>>>>>> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
>>>> further
>>>>>> develpment of backup/restore is blocked - the backup/restore is a key
>>>>>> enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
>>>>>> another round of vote.
>>>>>> 
>>>>>> 
>>>>> IMO, not a blocker since it ancillary function but something we should
>>>> get
>>>>> in.
>>>>> 
>>>>> 
>>>>> 
>>>>>> (3). HBASE-15179 Offheap
>>>>>> - This is another important feature that I think it is MUST for 2.0.
>>>>> Since
>>>>>> stack works closely with Intel developers, he has some insight on
>>> this
>>>>>> project: "Intel are betting on this. Alibaba are using the offheap
>>> read
>>>>>> path and interested in write path too. This work is still very much
>>> up
>>>> in
>>>>>> the air and being worked out as we go (especially the Y! Israel
>>>> inmemory
>>>>>> compaction component).  It is a little shakey dependent on mslab
>>>> pooling,
>>>>>> blockcache being on by default, async wal being default, and then
>>>>> dependent
>>>>>> on lots of perf and ITBLL testing."
>>>>>> 
>>>>>> 
>>>>> The above is from a private, hurried email that does not do the current
>>>>> state justice.
>>>>> 
>>>>> Ram and Anoop are the authorities on offheaping and have been keeping
>>> up
>>>>> their state in an associated doc. The lads might be persuaded do a
>>>> summary
>>>>> of current state here.
>>>>> 
>>>>> Ditto for inmemory compaction. Our Anastasia and Eshcar are working
>>> hard
>>>> at
>>>>> the moment doing laste pieces and perf testing. Maybe we can a status
>>>>> dumped here in dev.
>>>>> 
>>>>> 
>>>>> 
>>>>>> (4). HBASE-16952 Protobuf3
>>>>>> - Good news, stack got the majority of work done already.  This is a
>>>> MUST
>>>>>> for 2.0 release.  Now we only have a small sub-task HBASE-16967
>>>>> (findbugs)
>>>>>> left.
>>>>>> 
>>>>>> 
>>>>> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
>>> to
>>>>> this list, also needed by offheaping.
>>>>> 
>>>>> Will be back after taking a look at current 2.0 list.
>>>>> 
>>>>> Agree that hbase2 needs to support hadoop3.
>>>>> 
>>>>> St.Ack
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> (5). HBASE-14070 Logical Clock
>>>>>> - This needs to be done before 2.0 release.  At this time, seems not
>>>>> making
>>>>>> much progress.
>>>>>> 
>>>>>> (6). HBASE-16833 JAVA Async Client
>>>>>> - A lot of progress was made by Duo and his associates.  We should be
>>>> in
>>>>> a
>>>>>> good shape in JAVA client when 2.0 release.
>>>>>> 
>>>>>> (7). HBASE-14850 C++ Async Client
>>>>>> - Another project that is making good progress.  I know some customer
>>>>> wants
>>>>>> this.  This is long overdue project.  Hopefully we will make those
>>>>> customer
>>>>>> happy in 2.0 release with a good performed C++ client.
>>>>>> 
>>>>>> Please let me know other on-going projects that needs to be in HBASE
>>>> 2.0
>>>>>> release (stack mentioned "logical file system tier", but I am not
>>> sure
>>>>>> whether it would make enough progress to have a realistic chance
>>> making
>>>>>> 2.0).  As I said at the beginning of this email, at some point soon,
>>> we
>>>>>> will be more picky and trim down the unfinished features in 2.0.
>>>>>> 
>>>>>> Happy Weekend!
>>>>>> Stephen
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Thanks,
>>>> Michael Antonov
>>>> 

Re: HBASE 2.0 release progress

[Thiruvel Thirumoolan <th...@yahoo-inc.com.INVALID>]

We would like to add the favored node enhancements as part of https://issues.apache.org/jira/browse/HBASE-15531 to 2.0. Most of the code should be new, there will be very less code changes to Master/AM etc.

-Thiruvel

On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John <an...@gmail.com> wrote:>-  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
up-to-date state of offheaping?

Off heaping the read path (HBASE-11425) this is completed some time
back.  As per my knowledge at least 2 users back ported this work to
1.x based version and even running in production. Am leaving it them
for giving details..

Write path off heaping HBASE-15179 is the umbrella issue.  As u can
see most of the sub tasks are done by now..  Mainly 2 more sub tasks
are yet to get committed.  Both are some what bigger sized also..
Patch is already available for them.  We hope it can be completed by
Nov end.

Some more off heaping work, (in compaction path etc) might be taken up
after write path work. Alibaba guys might be joining with that.  Had
some discuss with them.. We will come up with jira and doc for that
before actual work begin.  But as Enis said ya all that can go in 2.x
(x>0) :-)

Just giving some high level status update. Thanks ..

-Anoop-


On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Enis, by your criteria it seems the log4j2 upgrade ticket should be
> considered for 2.0 as well: HBASE-12341.
>
> On Monday, November 14, 2016, Enis Söztutar <en...@gmail.com> wrote:
>
>> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
>> simple test:
>>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
>> not make into 2.0.
>>
>> It is a simple test, but affectively validates whether the feature is
>> client-visible and have backwards compatibility implications. If it is ok
>> to bring the feature in for 2.1 or later, it means that the release should
>> never block on it. Otherwise, we should at least have the API / Client
>> visible parts of it in 2.0 if not the full thing.
>>
>> Looking at the features under discussion through this lens reveals easier
>> decision points I think. A couple of examples:
>>  - Java async client / C++ Client: It is independent work, can come in 2.1
>> or 2.2, etc. Whenever it is completed.
>>  - Offheaping : transparent ti clients, so can come in incrementally in
>> 2.0, 2.1, 2.2, etc.
>>
>> On the other hand, 1.x releases have been going on for >1.5 years, and will
>> likely go on for another year at least. So the choices (dependency, APIs,
>> etc) that we make now for 2.0 will likely live for >2 years. In that
>> respect I would rather be a bit more aggressive in terms of dropping
>> support for older stuff and updating wherever we can. For example require
>> Hadoop-2.6+, Java-8, etc.
>>
>> Enis
>>
>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
>> <javascript:;>>
>> wrote:
>>
>> > Great to see progress here.
>> >
>> > As I'm reading through the list, here're some high-level questions I
>> have:
>> >
>> >  -  Regarding the work on new AssignmentManager - any notes on
>> > perf/stability testing? Are you guys running tips of master branch
>> through
>> > ITBLL setup?
>> >  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
>> current
>> > up-to-date state of offheaping?
>> >  -  Logical clock - at this point is it more like a nice to have feature
>> > than "need to be done before 2.0"? What are the features blocked or
>> > affected by lack thereof?
>> >
>> > -Mikhail
>> >
>> > On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
>> wrote:
>> >
>> > > Thanks for the writeup Stephen.
>> > >
>> > > See below.
>> > >
>> > > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
>> syuanjiangdev@gmail.com <javascript:;>>
>> > > wrote:
>> > >
>> > > > Hello, fellow HBASE developers,
>> > > >
>> > > > We are making progress towards HBASE 2.0 releases.  I am using the
>> > > > following queries to search for on-going HBASE 2.0 feature work items
>> > > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
>> > AND
>> > > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
>> > > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
>> have
>> > > > 247!  That is a lot.  At some time in near future, we definitely need
>> > to
>> > > > trim down the list; otherwise, 2.0 will never be released.
>> > > >
>> > > >
>> > > Agree. Suggest we all do our own review but at a certain stage, it is
>> up
>> > to
>> > > the RMs to make a call on what shape of 2.0 will be.
>> > >
>> > >
>> > >
>> > > > For now, Matteo and I are tracking some big projects that are
>> on-going:
>> > > >
>> > > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
>> > > > - This is a blocker to have branch-2 cut.  In the past few weeks, we
>> > made
>> > > > good progress and majority of implementation is done.  The patches
>> are
>> > > > under review and testing.  Matto is drafting a document for review.
>> > > >
>> > > > (2). HBASE-14414 Backup/Restore Phase 3
>> > > > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
>> > patches
>> > > > was discussed and reviewed from the community (see
>> > > > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
>> the
>> > > > long
>> > > > discussion); and all feedback were taken care of in the latest patch.
>> > > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
>> > further
>> > > > develpment of backup/restore is blocked - the backup/restore is a key
>> > > > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
>> > > > another round of vote.
>> > > >
>> > > >
>> > > IMO, not a blocker since it ancillary function but something we should
>> > get
>> > > in.
>> > >
>> > >
>> > >
>> > > > (3). HBASE-15179 Offheap
>> > > > - This is another important feature that I think it is MUST for 2.0.
>> > > Since
>> > > > stack works closely with Intel developers, he has some insight on
>> this
>> > > > project: "Intel are betting on this. Alibaba are using the offheap
>> read
>> > > > path and interested in write path too. This work is still very much
>> up
>> > in
>> > > > the air and being worked out as we go (especially the Y! Israel
>> > inmemory
>> > > > compaction component).  It is a little shakey dependent on mslab
>> > pooling,
>> > > > blockcache being on by default, async wal being default, and then
>> > > dependent
>> > > > on lots of perf and ITBLL testing."
>> > > >
>> > > >
>> > > The above is from a private, hurried email that does not do the current
>> > > state justice.
>> > >
>> > > Ram and Anoop are the authorities on offheaping and have been keeping
>> up
>> > > their state in an associated doc. The lads might be persuaded do a
>> > summary
>> > > of current state here.
>> > >
>> > > Ditto for inmemory compaction. Our Anastasia and Eshcar are working
>> hard
>> > at
>> > > the moment doing laste pieces and perf testing. Maybe we can a status
>> > > dumped here in dev.
>> > >
>> > >
>> > >
>> > > > (4). HBASE-16952 Protobuf3
>> > > > - Good news, stack got the majority of work done already.  This is a
>> > MUST
>> > > > for 2.0 release.  Now we only have a small sub-task HBASE-16967
>> > > (findbugs)
>> > > > left.
>> > > >
>> > > >
>> > > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
>> to
>> > > this list, also needed by offheaping.
>> > >
>> > > Will be back after taking a look at current 2.0 list.
>> > >
>> > > Agree that hbase2 needs to support hadoop3.
>> > >
>> > > St.Ack
>> > >
>> > >
>> > >
>> > >
>> > > > (5). HBASE-14070 Logical Clock
>> > > > - This needs to be done before 2.0 release.  At this time, seems not
>> > > making
>> > > > much progress.
>> > > >
>> > > > (6). HBASE-16833 JAVA Async Client
>> > > > - A lot of progress was made by Duo and his associates.  We should be
>> > in
>> > > a
>> > > > good shape in JAVA client when 2.0 release.
>> > > >
>> > > > (7). HBASE-14850 C++ Async Client
>> > > > - Another project that is making good progress.  I know some customer
>> > > wants
>> > > > this.  This is long overdue project.  Hopefully we will make those
>> > > customer
>> > > > happy in 2.0 release with a good performed C++ client.
>> > > >
>> > > > Please let me know other on-going projects that needs to be in HBASE
>> > 2.0
>> > > > release (stack mentioned "logical file system tier", but I am not
>> sure
>> > > > whether it would make enough progress to have a realistic chance
>> making
>> > > > 2.0).  As I said at the beginning of this email, at some point soon,
>> we
>> > > > will be more picky and trim down the unfinished features in 2.0.
>> > > >
>> > > > Happy Weekend!
>> > > > Stephen
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Michael Antonov
>> >
>>

Re: HBASE 2.0 release progress

[Anoop John <an...@gmail.com>]

>-  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
up-to-date state of offheaping?

Off heaping the read path (HBASE-11425) this is completed some time
back.  As per my knowledge at least 2 users back ported this work to
1.x based version and even running in production. Am leaving it them
for giving details..

Write path off heaping HBASE-15179 is the umbrella issue.  As u can
see most of the sub tasks are done by now..  Mainly 2 more sub tasks
are yet to get committed.  Both are some what bigger sized also..
Patch is already available for them.   We hope it can be completed by
Nov end.

Some more off heaping work, (in compaction path etc) might be taken up
after write path work. Alibaba guys might be joining with that.  Had
some discuss with them.. We will come up with jira and doc for that
before actual work begin.  But as Enis said ya all that can go in 2.x
(x>0) :-)

Just giving some high level status update. Thanks ..

-Anoop-


On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Enis, by your criteria it seems the log4j2 upgrade ticket should be
> considered for 2.0 as well: HBASE-12341.
>
> On Monday, November 14, 2016, Enis Söztutar <en...@gmail.com> wrote:
>
>> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
>> simple test:
>>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
>> not make into 2.0.
>>
>> It is a simple test, but affectively validates whether the feature is
>> client-visible and have backwards compatibility implications. If it is ok
>> to bring the feature in for 2.1 or later, it means that the release should
>> never block on it. Otherwise, we should at least have the API / Client
>> visible parts of it in 2.0 if not the full thing.
>>
>> Looking at the features under discussion through this lens reveals easier
>> decision points I think. A couple of examples:
>>  - Java async client / C++ Client: It is independent work, can come in 2.1
>> or 2.2, etc. Whenever it is completed.
>>  - Offheaping : transparent ti clients, so can come in incrementally in
>> 2.0, 2.1, 2.2, etc.
>>
>> On the other hand, 1.x releases have been going on for >1.5 years, and will
>> likely go on for another year at least. So the choices (dependency, APIs,
>> etc) that we make now for 2.0 will likely live for >2 years. In that
>> respect I would rather be a bit more aggressive in terms of dropping
>> support for older stuff and updating wherever we can. For example require
>> Hadoop-2.6+, Java-8, etc.
>>
>> Enis
>>
>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
>> <javascript:;>>
>> wrote:
>>
>> > Great to see progress here.
>> >
>> > As I'm reading through the list, here're some high-level questions I
>> have:
>> >
>> >  -  Regarding the work on new AssignmentManager - any notes on
>> > perf/stability testing? Are you guys running tips of master branch
>> through
>> > ITBLL setup?
>> >  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
>> current
>> > up-to-date state of offheaping?
>> >  -  Logical clock - at this point is it more like a nice to have feature
>> > than "need to be done before 2.0"? What are the features blocked or
>> > affected by lack thereof?
>> >
>> > -Mikhail
>> >
>> > On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
>> wrote:
>> >
>> > > Thanks for the writeup Stephen.
>> > >
>> > > See below.
>> > >
>> > > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
>> syuanjiangdev@gmail.com <javascript:;>>
>> > > wrote:
>> > >
>> > > > Hello, fellow HBASE developers,
>> > > >
>> > > > We are making progress towards HBASE 2.0 releases.  I am using the
>> > > > following queries to search for on-going HBASE 2.0 feature work items
>> > > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
>> > AND
>> > > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
>> > > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
>> have
>> > > > 247!  That is a lot.  At some time in near future, we definitely need
>> > to
>> > > > trim down the list; otherwise, 2.0 will never be released.
>> > > >
>> > > >
>> > > Agree. Suggest we all do our own review but at a certain stage, it is
>> up
>> > to
>> > > the RMs to make a call on what shape of 2.0 will be.
>> > >
>> > >
>> > >
>> > > > For now, Matteo and I are tracking some big projects that are
>> on-going:
>> > > >
>> > > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
>> > > > - This is a blocker to have branch-2 cut.  In the past few weeks, we
>> > made
>> > > > good progress and majority of implementation is done.  The patches
>> are
>> > > > under review and testing.  Matto is drafting a document for review.
>> > > >
>> > > > (2). HBASE-14414 Backup/Restore Phase 3
>> > > > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
>> > patches
>> > > > was discussed and reviewed from the community (see
>> > > > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
>> the
>> > > > long
>> > > > discussion); and all feedback were taken care of in the latest patch.
>> > > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
>> > further
>> > > > develpment of backup/restore is blocked - the backup/restore is a key
>> > > > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
>> > > > another round of vote.
>> > > >
>> > > >
>> > > IMO, not a blocker since it ancillary function but something we should
>> > get
>> > > in.
>> > >
>> > >
>> > >
>> > > > (3). HBASE-15179 Offheap
>> > > > - This is another important feature that I think it is MUST for 2.0.
>> > > Since
>> > > > stack works closely with Intel developers, he has some insight on
>> this
>> > > > project: "Intel are betting on this. Alibaba are using the offheap
>> read
>> > > > path and interested in write path too. This work is still very much
>> up
>> > in
>> > > > the air and being worked out as we go (especially the Y! Israel
>> > inmemory
>> > > > compaction component).  It is a little shakey dependent on mslab
>> > pooling,
>> > > > blockcache being on by default, async wal being default, and then
>> > > dependent
>> > > > on lots of perf and ITBLL testing."
>> > > >
>> > > >
>> > > The above is from a private, hurried email that does not do the current
>> > > state justice.
>> > >
>> > > Ram and Anoop are the authorities on offheaping and have been keeping
>> up
>> > > their state in an associated doc. The lads might be persuaded do a
>> > summary
>> > > of current state here.
>> > >
>> > > Ditto for inmemory compaction. Our Anastasia and Eshcar are working
>> hard
>> > at
>> > > the moment doing laste pieces and perf testing. Maybe we can a status
>> > > dumped here in dev.
>> > >
>> > >
>> > >
>> > > > (4). HBASE-16952 Protobuf3
>> > > > - Good news, stack got the majority of work done already.  This is a
>> > MUST
>> > > > for 2.0 release.  Now we only have a small sub-task HBASE-16967
>> > > (findbugs)
>> > > > left.
>> > > >
>> > > >
>> > > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
>> to
>> > > this list, also needed by offheaping.
>> > >
>> > > Will be back after taking a look at current 2.0 list.
>> > >
>> > > Agree that hbase2 needs to support hadoop3.
>> > >
>> > > St.Ack
>> > >
>> > >
>> > >
>> > >
>> > > > (5). HBASE-14070 Logical Clock
>> > > > - This needs to be done before 2.0 release.  At this time, seems not
>> > > making
>> > > > much progress.
>> > > >
>> > > > (6). HBASE-16833 JAVA Async Client
>> > > > - A lot of progress was made by Duo and his associates.  We should be
>> > in
>> > > a
>> > > > good shape in JAVA client when 2.0 release.
>> > > >
>> > > > (7). HBASE-14850 C++ Async Client
>> > > > - Another project that is making good progress.  I know some customer
>> > > wants
>> > > > this.  This is long overdue project.  Hopefully we will make those
>> > > customer
>> > > > happy in 2.0 release with a good performed C++ client.
>> > > >
>> > > > Please let me know other on-going projects that needs to be in HBASE
>> > 2.0
>> > > > release (stack mentioned "logical file system tier", but I am not
>> sure
>> > > > whether it would make enough progress to have a realistic chance
>> making
>> > > > 2.0).  As I said at the beginning of this email, at some point soon,
>> we
>> > > > will be more picky and trim down the unfinished features in 2.0.
>> > > >
>> > > > Happy Weekend!
>> > > > Stephen
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Michael Antonov
>> >
>>

Re: HBASE 2.0 release progress

[Nick Dimiduk <nd...@gmail.com>]

Enis, by your criteria it seems the log4j2 upgrade ticket should be
considered for 2.0 as well: HBASE-12341.

On Monday, November 14, 2016, Enis Söztutar <en...@gmail.com> wrote:

> Another way to look at whether a feature is a "blocker" or not for 2.0 is a
> simple test:
>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
> not make into 2.0.
>
> It is a simple test, but affectively validates whether the feature is
> client-visible and have backwards compatibility implications. If it is ok
> to bring the feature in for 2.1 or later, it means that the release should
> never block on it. Otherwise, we should at least have the API / Client
> visible parts of it in 2.0 if not the full thing.
>
> Looking at the features under discussion through this lens reveals easier
> decision points I think. A couple of examples:
>  - Java async client / C++ Client: It is independent work, can come in 2.1
> or 2.2, etc. Whenever it is completed.
>  - Offheaping : transparent ti clients, so can come in incrementally in
> 2.0, 2.1, 2.2, etc.
>
> On the other hand, 1.x releases have been going on for >1.5 years, and will
> likely go on for another year at least. So the choices (dependency, APIs,
> etc) that we make now for 2.0 will likely live for >2 years. In that
> respect I would rather be a bit more aggressive in terms of dropping
> support for older stuff and updating wherever we can. For example require
> Hadoop-2.6+, Java-8, etc.
>
> Enis
>
> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <olorinbant@gmail.com
> <javascript:;>>
> wrote:
>
> > Great to see progress here.
> >
> > As I'm reading through the list, here're some high-level questions I
> have:
> >
> >  -  Regarding the work on new AssignmentManager - any notes on
> > perf/stability testing? Are you guys running tips of master branch
> through
> > ITBLL setup?
> >  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the
> current
> > up-to-date state of offheaping?
> >  -  Logical clock - at this point is it more like a nice to have feature
> > than "need to be done before 2.0"? What are the features blocked or
> > affected by lack thereof?
> >
> > -Mikhail
> >
> > On Sun, Nov 13, 2016 at 3:57 PM, Stack <stack@duboce.net <javascript:;>>
> wrote:
> >
> > > Thanks for the writeup Stephen.
> > >
> > > See below.
> > >
> > > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <
> syuanjiangdev@gmail.com <javascript:;>>
> > > wrote:
> > >
> > > > Hello, fellow HBASE developers,
> > > >
> > > > We are making progress towards HBASE 2.0 releases.  I am using the
> > > > following queries to search for on-going HBASE 2.0 feature work items
> > > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
> > AND
> > > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we
> have
> > > > 247!  That is a lot.  At some time in near future, we definitely need
> > to
> > > > trim down the list; otherwise, 2.0 will never be released.
> > > >
> > > >
> > > Agree. Suggest we all do our own review but at a certain stage, it is
> up
> > to
> > > the RMs to make a call on what shape of 2.0 will be.
> > >
> > >
> > >
> > > > For now, Matteo and I are tracking some big projects that are
> on-going:
> > > >
> > > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > > > - This is a blocker to have branch-2 cut.  In the past few weeks, we
> > made
> > > > good progress and majority of implementation is done.  The patches
> are
> > > > under review and testing.  Matto is drafting a document for review.
> > > >
> > > > (2). HBASE-14414 Backup/Restore Phase 3
> > > > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
> > patches
> > > > was discussed and reviewed from the community (see
> > > > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for
> the
> > > > long
> > > > discussion); and all feedback were taken care of in the latest patch.
> > > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
> > further
> > > > develpment of backup/restore is blocked - the backup/restore is a key
> > > > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > > > another round of vote.
> > > >
> > > >
> > > IMO, not a blocker since it ancillary function but something we should
> > get
> > > in.
> > >
> > >
> > >
> > > > (3). HBASE-15179 Offheap
> > > > - This is another important feature that I think it is MUST for 2.0.
> > > Since
> > > > stack works closely with Intel developers, he has some insight on
> this
> > > > project: "Intel are betting on this. Alibaba are using the offheap
> read
> > > > path and interested in write path too. This work is still very much
> up
> > in
> > > > the air and being worked out as we go (especially the Y! Israel
> > inmemory
> > > > compaction component).  It is a little shakey dependent on mslab
> > pooling,
> > > > blockcache being on by default, async wal being default, and then
> > > dependent
> > > > on lots of perf and ITBLL testing."
> > > >
> > > >
> > > The above is from a private, hurried email that does not do the current
> > > state justice.
> > >
> > > Ram and Anoop are the authorities on offheaping and have been keeping
> up
> > > their state in an associated doc. The lads might be persuaded do a
> > summary
> > > of current state here.
> > >
> > > Ditto for inmemory compaction. Our Anastasia and Eshcar are working
> hard
> > at
> > > the moment doing laste pieces and perf testing. Maybe we can a status
> > > dumped here in dev.
> > >
> > >
> > >
> > > > (4). HBASE-16952 Protobuf3
> > > > - Good news, stack got the majority of work done already.  This is a
> > MUST
> > > > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> > > (findbugs)
> > > > left.
> > > >
> > > >
> > > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL
> to
> > > this list, also needed by offheaping.
> > >
> > > Will be back after taking a look at current 2.0 list.
> > >
> > > Agree that hbase2 needs to support hadoop3.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > (5). HBASE-14070 Logical Clock
> > > > - This needs to be done before 2.0 release.  At this time, seems not
> > > making
> > > > much progress.
> > > >
> > > > (6). HBASE-16833 JAVA Async Client
> > > > - A lot of progress was made by Duo and his associates.  We should be
> > in
> > > a
> > > > good shape in JAVA client when 2.0 release.
> > > >
> > > > (7). HBASE-14850 C++ Async Client
> > > > - Another project that is making good progress.  I know some customer
> > > wants
> > > > this.  This is long overdue project.  Hopefully we will make those
> > > customer
> > > > happy in 2.0 release with a good performed C++ client.
> > > >
> > > > Please let me know other on-going projects that needs to be in HBASE
> > 2.0
> > > > release (stack mentioned "logical file system tier", but I am not
> sure
> > > > whether it would make enough progress to have a realistic chance
> making
> > > > 2.0).  As I said at the beginning of this email, at some point soon,
> we
> > > > will be more picky and trim down the unfinished features in 2.0.
> > > >
> > > > Happy Weekend!
> > > > Stephen
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
> >
>

Re: HBASE 2.0 release progress

[Enis Söztutar <en...@gmail.com>]

Another way to look at whether a feature is a "blocker" or not for 2.0 is a
simple test:
 - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
not make into 2.0.

It is a simple test, but affectively validates whether the feature is
client-visible and have backwards compatibility implications. If it is ok
to bring the feature in for 2.1 or later, it means that the release should
never block on it. Otherwise, we should at least have the API / Client
visible parts of it in 2.0 if not the full thing.

Looking at the features under discussion through this lens reveals easier
decision points I think. A couple of examples:
 - Java async client / C++ Client: It is independent work, can come in 2.1
or 2.2, etc. Whenever it is completed.
 - Offheaping : transparent ti clients, so can come in incrementally in
2.0, 2.1, 2.2, etc.

On the other hand, 1.x releases have been going on for >1.5 years, and will
likely go on for another year at least. So the choices (dependency, APIs,
etc) that we make now for 2.0 will likely live for >2 years. In that
respect I would rather be a bit more aggressive in terms of dropping
support for older stuff and updating wherever we can. For example require
Hadoop-2.6+, Java-8, etc.

Enis

On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov <ol...@gmail.com>
wrote:

> Great to see progress here.
>
> As I'm reading through the list, here're some high-level questions I have:
>
>  -  Regarding the work on new AssignmentManager - any notes on
> perf/stability testing? Are you guys running tips of master branch through
> ITBLL setup?
>  -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
> up-to-date state of offheaping?
>  -  Logical clock - at this point is it more like a nice to have feature
> than "need to be done before 2.0"? What are the features blocked or
> affected by lack thereof?
>
> -Mikhail
>
> On Sun, Nov 13, 2016 at 3:57 PM, Stack <st...@duboce.net> wrote:
>
> > Thanks for the writeup Stephen.
> >
> > See below.
> >
> > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
> > wrote:
> >
> > > Hello, fellow HBASE developers,
> > >
> > > We are making progress towards HBASE 2.0 releases.  I am using the
> > > following queries to search for on-going HBASE 2.0 feature work items
> > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0)
> AND
> > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> > > 247!  That is a lot.  At some time in near future, we definitely need
> to
> > > trim down the list; otherwise, 2.0 will never be released.
> > >
> > >
> > Agree. Suggest we all do our own review but at a certain stage, it is up
> to
> > the RMs to make a call on what shape of 2.0 will be.
> >
> >
> >
> > > For now, Matteo and I are tracking some big projects that are on-going:
> > >
> > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > > - This is a blocker to have branch-2 cut.  In the past few weeks, we
> made
> > > good progress and majority of implementation is done.  The patches are
> > > under review and testing.  Matto is drafting a document for review.
> > >
> > > (2). HBASE-14414 Backup/Restore Phase 3
> > > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123
> patches
> > > was discussed and reviewed from the community (see
> > > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> > > long
> > > discussion); and all feedback were taken care of in the latest patch.
> > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the
> further
> > > develpment of backup/restore is blocked - the backup/restore is a key
> > > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > > another round of vote.
> > >
> > >
> > IMO, not a blocker since it ancillary function but something we should
> get
> > in.
> >
> >
> >
> > > (3). HBASE-15179 Offheap
> > > - This is another important feature that I think it is MUST for 2.0.
> > Since
> > > stack works closely with Intel developers, he has some insight on this
> > > project: "Intel are betting on this. Alibaba are using the offheap read
> > > path and interested in write path too. This work is still very much up
> in
> > > the air and being worked out as we go (especially the Y! Israel
> inmemory
> > > compaction component).  It is a little shakey dependent on mslab
> pooling,
> > > blockcache being on by default, async wal being default, and then
> > dependent
> > > on lots of perf and ITBLL testing."
> > >
> > >
> > The above is from a private, hurried email that does not do the current
> > state justice.
> >
> > Ram and Anoop are the authorities on offheaping and have been keeping up
> > their state in an associated doc. The lads might be persuaded do a
> summary
> > of current state here.
> >
> > Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard
> at
> > the moment doing laste pieces and perf testing. Maybe we can a status
> > dumped here in dev.
> >
> >
> >
> > > (4). HBASE-16952 Protobuf3
> > > - Good news, stack got the majority of work done already.  This is a
> MUST
> > > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> > (findbugs)
> > > left.
> > >
> > >
> > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to
> > this list, also needed by offheaping.
> >
> > Will be back after taking a look at current 2.0 list.
> >
> > Agree that hbase2 needs to support hadoop3.
> >
> > St.Ack
> >
> >
> >
> >
> > > (5). HBASE-14070 Logical Clock
> > > - This needs to be done before 2.0 release.  At this time, seems not
> > making
> > > much progress.
> > >
> > > (6). HBASE-16833 JAVA Async Client
> > > - A lot of progress was made by Duo and his associates.  We should be
> in
> > a
> > > good shape in JAVA client when 2.0 release.
> > >
> > > (7). HBASE-14850 C++ Async Client
> > > - Another project that is making good progress.  I know some customer
> > wants
> > > this.  This is long overdue project.  Hopefully we will make those
> > customer
> > > happy in 2.0 release with a good performed C++ client.
> > >
> > > Please let me know other on-going projects that needs to be in HBASE
> 2.0
> > > release (stack mentioned "logical file system tier", but I am not sure
> > > whether it would make enough progress to have a realistic chance making
> > > 2.0).  As I said at the beginning of this email, at some point soon, we
> > > will be more picky and trim down the unfinished features in 2.0.
> > >
> > > Happy Weekend!
> > > Stephen
> > >
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>

Re: HBASE 2.0 release progress

[Mikhail Antonov <ol...@gmail.com>]

Great to see progress here.

As I'm reading through the list, here're some high-level questions I have:

 -  Regarding the work on new AssignmentManager - any notes on
perf/stability testing? Are you guys running tips of master branch through
ITBLL setup?
 -  Anyone (Ram, Anoop?) wants to post a high-level writeup on the current
up-to-date state of offheaping?
 -  Logical clock - at this point is it more like a nice to have feature
than "need to be done before 2.0"? What are the features blocked or
affected by lack thereof?

-Mikhail

On Sun, Nov 13, 2016 at 3:57 PM, Stack <st...@duboce.net> wrote:

> Thanks for the writeup Stephen.
>
> See below.
>
> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
>
> > Hello, fellow HBASE developers,
> >
> > We are making progress towards HBASE 2.0 releases.  I am using the
> > following queries to search for on-going HBASE 2.0 feature work items
> > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> > 247!  That is a lot.  At some time in near future, we definitely need to
> > trim down the list; otherwise, 2.0 will never be released.
> >
> >
> Agree. Suggest we all do our own review but at a certain stage, it is up to
> the RMs to make a call on what shape of 2.0 will be.
>
>
>
> > For now, Matteo and I are tracking some big projects that are on-going:
> >
> > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> > good progress and majority of implementation is done.  The patches are
> > under review and testing.  Matto is drafting a document for review.
> >
> > (2). HBASE-14414 Backup/Restore Phase 3
> > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> > was discussed and reviewed from the community (see
> > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> > long
> > discussion); and all feedback were taken care of in the latest patch.
> > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> > develpment of backup/restore is blocked - the backup/restore is a key
> > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > another round of vote.
> >
> >
> IMO, not a blocker since it ancillary function but something we should get
> in.
>
>
>
> > (3). HBASE-15179 Offheap
> > - This is another important feature that I think it is MUST for 2.0.
> Since
> > stack works closely with Intel developers, he has some insight on this
> > project: "Intel are betting on this. Alibaba are using the offheap read
> > path and interested in write path too. This work is still very much up in
> > the air and being worked out as we go (especially the Y! Israel inmemory
> > compaction component).  It is a little shakey dependent on mslab pooling,
> > blockcache being on by default, async wal being default, and then
> dependent
> > on lots of perf and ITBLL testing."
> >
> >
> The above is from a private, hurried email that does not do the current
> state justice.
>
> Ram and Anoop are the authorities on offheaping and have been keeping up
> their state in an associated doc. The lads might be persuaded do a summary
> of current state here.
>
> Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard at
> the moment doing laste pieces and perf testing. Maybe we can a status
> dumped here in dev.
>
>
>
> > (4). HBASE-16952 Protobuf3
> > - Good news, stack got the majority of work done already.  This is a MUST
> > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> (findbugs)
> > left.
> >
> >
> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to
> this list, also needed by offheaping.
>
> Will be back after taking a look at current 2.0 list.
>
> Agree that hbase2 needs to support hadoop3.
>
> St.Ack
>
>
>
>
> > (5). HBASE-14070 Logical Clock
> > - This needs to be done before 2.0 release.  At this time, seems not
> making
> > much progress.
> >
> > (6). HBASE-16833 JAVA Async Client
> > - A lot of progress was made by Duo and his associates.  We should be in
> a
> > good shape in JAVA client when 2.0 release.
> >
> > (7). HBASE-14850 C++ Async Client
> > - Another project that is making good progress.  I know some customer
> wants
> > this.  This is long overdue project.  Hopefully we will make those
> customer
> > happy in 2.0 release with a good performed C++ client.
> >
> > Please let me know other on-going projects that needs to be in HBASE 2.0
> > release (stack mentioned "logical file system tier", but I am not sure
> > whether it would make enough progress to have a realistic chance making
> > 2.0).  As I said at the beginning of this email, at some point soon, we
> > will be more picky and trim down the unfinished features in 2.0.
> >
> > Happy Weekend!
> > Stephen
> >
>



-- 
Thanks,
Michael Antonov

Re: HBASE 2.0 release progress

[Stack <st...@duboce.net>]

Thanks for the writeup Stephen.

See below.

On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <sy...@gmail.com>
wrote:

> Hello, fellow HBASE developers,
>
> We are making progress towards HBASE 2.0 releases.  I am using the
> following queries to search for on-going HBASE 2.0 feature work items
> (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> 247!  That is a lot.  At some time in near future, we definitely need to
> trim down the list; otherwise, 2.0 will never be released.
>
>
Agree. Suggest we all do our own review but at a certain stage, it is up to
the RMs to make a call on what shape of 2.0 will be.



> For now, Matteo and I are tracking some big projects that are on-going:
>
> (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> good progress and majority of implementation is done.  The patches are
> under review and testing.  Matto is drafting a document for review.
>
> (2). HBASE-14414 Backup/Restore Phase 3
> - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> was discussed and reviewed from the community (see
> http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> long
> discussion); and all feedback were taken care of in the latest patch.
> Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> develpment of backup/restore is blocked - the backup/restore is a key
> enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> another round of vote.
>
>
IMO, not a blocker since it ancillary function but something we should get
in.



> (3). HBASE-15179 Offheap
> - This is another important feature that I think it is MUST for 2.0.  Since
> stack works closely with Intel developers, he has some insight on this
> project: "Intel are betting on this. Alibaba are using the offheap read
> path and interested in write path too. This work is still very much up in
> the air and being worked out as we go (especially the Y! Israel inmemory
> compaction component).  It is a little shakey dependent on mslab pooling,
> blockcache being on by default, async wal being default, and then dependent
> on lots of perf and ITBLL testing."
>
>
The above is from a private, hurried email that does not do the current
state justice.

Ram and Anoop are the authorities on offheaping and have been keeping up
their state in an associated doc. The lads might be persuaded do a summary
of current state here.

Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard at
the moment doing laste pieces and perf testing. Maybe we can a status
dumped here in dev.



> (4). HBASE-16952 Protobuf3
> - Good news, stack got the majority of work done already.  This is a MUST
> for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
> left.
>
>
To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to
this list, also needed by offheaping.

Will be back after taking a look at current 2.0 list.

Agree that hbase2 needs to support hadoop3.

St.Ack




> (5). HBASE-14070 Logical Clock
> - This needs to be done before 2.0 release.  At this time, seems not making
> much progress.
>
> (6). HBASE-16833 JAVA Async Client
> - A lot of progress was made by Duo and his associates.  We should be in a
> good shape in JAVA client when 2.0 release.
>
> (7). HBASE-14850 C++ Async Client
> - Another project that is making good progress.  I know some customer wants
> this.  This is long overdue project.  Hopefully we will make those customer
> happy in 2.0 release with a good performed C++ client.
>
> Please let me know other on-going projects that needs to be in HBASE 2.0
> release (stack mentioned "logical file system tier", but I am not sure
> whether it would make enough progress to have a realistic chance making
> 2.0).  As I said at the beginning of this email, at some point soon, we
> will be more picky and trim down the unfinished features in 2.0.
>
> Happy Weekend!
> Stephen
>