You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Sree Vaddi <sr...@yahoo.com.INVALID> on 2019/06/27 23:28:22 UTC
Re: Please help to review patch of the HDFS-14585 (backport Byte
What are your challenges upgrading to Hadoop 3 ?
Did you verify if your problem statement valid, if you use Ozone API ?
Sent from Yahoo Mail on Android
On Thu, Jun 27, 2019 at 9:20 AM, OpenInx<op...@gmail.com> wrote: Dear HDFS Team:
I'm apache HBase member and working on HBASE-21879 [1]: it's a feature
which help to optimize the
HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
HDFS block into offheap
ByteBuffers and it had great performance improvement in our benchmark test
[2]. While the offheap
HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
as: ByteBuffer stream
read and ByteBuffer positional read.
The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3]
for Hadoop 3.3.0 version. but
the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
upgrade the dependency to
Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 so
that in the next hadoop
2.9.3 can include it , and our Apache HBase can upgrade the dependency to
2.9.3.
Now all the patches of HBASE-21879 have been merged into HBase master
branch. If we cannot upgrade to
the Hadoop which include the ByteBuffer pread implementation, then I
believe almost all of work from HBASE-21879
will worth nothing because we still read heap block and still high GC
pressure in some special case ( such as
cacheHitRatio is not very high). That will be quite frustrating for us.
Actually, we have been pushing the backport for the past few weeks. our
XiaoMi HDFS team have prepared the
backport patches, but seems the guys we ping didn't have the bandwidth to
help review the patches. we
expect the backport can be included in Hadoop 2.9.3 so that our upstream
can benefit from this as soon as
possible.
If any HDFS committer or PMC have some bandwidth to review the backport
patches, please give us a hand.
The Apache HBase team really need your help.
The ByteBuffer pread backport issues are:
https://issues.apache.org/jira/browse/HDFS-14585
https://issues.apache.org/jira/browse/HDFS-14483
Thank you very much.
[1]. https://issues.apache.org/jira/browse/HBASE-21879
[2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
[3]. https://issues.apache.org/jira/browse/HDFS-3246
Re: Please help to review patch of the HDFS-14585 (backport Byte
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
Thanks for your reply Sree Vaddi. As you said, in our internal hadoop
branch we have already backported these changes, and also it is fine for us
to upgrade to hadoop 3, in fact we have already started to test hadoop
3.1.x in our company.
But here we are talking about the open source version of hbase being
affected by these changes. For the open source version, I do not think we
can drop all the hadoop 2.x support for hbase 2.x releases. We have to
follow some rules, for example, only drop hadoop support when there are
CVEs and the hadoop team will not fix it for a minor release line. And when
changing the major versions, we can drop some legacy hadoop support, for
example, when releasing hbase 3.x, probably we could drop all the support
for hadoop 2.x.
Thanks. Regards
OpenInx <op...@gmail.com> 于2019年6月28日周五 上午10:58写道：
> > What are your challenges upgrading to Hadoop 3 ?
>
> It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams
> will need some time to test the stability
> and performance, so I think the HBase will not upgrade to hadoop 3 so quick
> in short time. Backporting the ByteBuffer
> pread to hadoop 2.9 is more suitable for now (it's a minor feature, should
> be OK to port in branch-2.9). I believe in future
> HBase release we will include the hadoop 3
>
> > Did you verify if your problem statement valid, if you use Ozone API ?
> We have some work about the object-store api [1]. but I'm not quite sure
> how will it affect the GC. will take a look about that.
>
> Thanks for your reply.
>
> [1]. https://issues.apache.org/jira/browse/HBASE-22149
> On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sr...@yahoo.com>
> wrote:
>
> > What are your challenges upgrading to Hadoop 3 ?
> >
> > Did you verify if your problem statement valid, if you use Ozone API ?
> >
> > Sent from Yahoo Mail on Android
> > <
> https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature
> >
> >
> > On Thu, Jun 27, 2019 at 9:20 AM, OpenInx
> > <op...@gmail.com> wrote:
> > Dear HDFS Team:
> >
> > I'm apache HBase member and working on HBASE-21879 [1]: it's a feature
> > which help to optimize the
> > HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
> > HDFS block into offheap
> > ByteBuffers and it had great performance improvement in our benchmark
> test
> > [2]. While the offheap
> > HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
> > as: ByteBuffer stream
> > read and ByteBuffer positional read.
> >
> > The problem is ByteBuffer pread interface: we've finished the
> HDFS-3246[3]
> > for Hadoop 3.3.0 version. but
> > the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
> > upgrade the dependency to
> > Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9
> so
> > that in the next hadoop
> > 2.9.3 can include it , and our Apache HBase can upgrade the dependency to
> > 2.9.3.
> >
> > Now all the patches of HBASE-21879 have been merged into HBase master
> > branch. If we cannot upgrade to
> > the Hadoop which include the ByteBuffer pread implementation, then I
> > believe almost all of work from HBASE-21879
> > will worth nothing because we still read heap block and still high GC
> > pressure in some special case ( such as
> > cacheHitRatio is not very high). That will be quite frustrating for us.
> >
> > Actually, we have been pushing the backport for the past few weeks. our
> > XiaoMi HDFS team have prepared the
> > backport patches, but seems the guys we ping didn't have the bandwidth to
> > help review the patches. we
> > expect the backport can be included in Hadoop 2.9.3 so that our upstream
> > can benefit from this as soon as
> > possible.
> >
> > If any HDFS committer or PMC have some bandwidth to review the backport
> > patches, please give us a hand.
> > The Apache HBase team really need your help.
> >
> > The ByteBuffer pread backport issues are:
> > https://issues.apache.org/jira/browse/HDFS-14585
> > https://issues.apache.org/jira/browse/HDFS-14483
> >
> > Thank you very much.
> >
> > [1]. https://issues.apache.org/jira/browse/HBASE-21879
> > [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
> > [3]. https://issues.apache.org/jira/browse/HDFS-3246
> >
> >
>
Re: Please help to review patch of the HDFS-14585 (backport Byte
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
Thanks for your reply Sree Vaddi. As you said, in our internal hadoop
branch we have already backported these changes, and also it is fine for us
to upgrade to hadoop 3, in fact we have already started to test hadoop
3.1.x in our company.
But here we are talking about the open source version of hbase being
affected by these changes. For the open source version, I do not think we
can drop all the hadoop 2.x support for hbase 2.x releases. We have to
follow some rules, for example, only drop hadoop support when there are
CVEs and the hadoop team will not fix it for a minor release line. And when
changing the major versions, we can drop some legacy hadoop support, for
example, when releasing hbase 3.x, probably we could drop all the support
for hadoop 2.x.
Thanks. Regards
OpenInx <op...@gmail.com> 于2019年6月28日周五 上午10:58写道：
> > What are your challenges upgrading to Hadoop 3 ?
>
> It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams
> will need some time to test the stability
> and performance, so I think the HBase will not upgrade to hadoop 3 so quick
> in short time. Backporting the ByteBuffer
> pread to hadoop 2.9 is more suitable for now (it's a minor feature, should
> be OK to port in branch-2.9). I believe in future
> HBase release we will include the hadoop 3
>
> > Did you verify if your problem statement valid, if you use Ozone API ?
> We have some work about the object-store api [1]. but I'm not quite sure
> how will it affect the GC. will take a look about that.
>
> Thanks for your reply.
>
> [1]. https://issues.apache.org/jira/browse/HBASE-22149
> On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sr...@yahoo.com>
> wrote:
>
> > What are your challenges upgrading to Hadoop 3 ?
> >
> > Did you verify if your problem statement valid, if you use Ozone API ?
> >
> > Sent from Yahoo Mail on Android
> > <
> https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature
> >
> >
> > On Thu, Jun 27, 2019 at 9:20 AM, OpenInx
> > <op...@gmail.com> wrote:
> > Dear HDFS Team:
> >
> > I'm apache HBase member and working on HBASE-21879 [1]: it's a feature
> > which help to optimize the
> > HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
> > HDFS block into offheap
> > ByteBuffers and it had great performance improvement in our benchmark
> test
> > [2]. While the offheap
> > HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
> > as: ByteBuffer stream
> > read and ByteBuffer positional read.
> >
> > The problem is ByteBuffer pread interface: we've finished the
> HDFS-3246[3]
> > for Hadoop 3.3.0 version. but
> > the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
> > upgrade the dependency to
> > Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9
> so
> > that in the next hadoop
> > 2.9.3 can include it , and our Apache HBase can upgrade the dependency to
> > 2.9.3.
> >
> > Now all the patches of HBASE-21879 have been merged into HBase master
> > branch. If we cannot upgrade to
> > the Hadoop which include the ByteBuffer pread implementation, then I
> > believe almost all of work from HBASE-21879
> > will worth nothing because we still read heap block and still high GC
> > pressure in some special case ( such as
> > cacheHitRatio is not very high). That will be quite frustrating for us.
> >
> > Actually, we have been pushing the backport for the past few weeks. our
> > XiaoMi HDFS team have prepared the
> > backport patches, but seems the guys we ping didn't have the bandwidth to
> > help review the patches. we
> > expect the backport can be included in Hadoop 2.9.3 so that our upstream
> > can benefit from this as soon as
> > possible.
> >
> > If any HDFS committer or PMC have some bandwidth to review the backport
> > patches, please give us a hand.
> > The Apache HBase team really need your help.
> >
> > The ByteBuffer pread backport issues are:
> > https://issues.apache.org/jira/browse/HDFS-14585
> > https://issues.apache.org/jira/browse/HDFS-14483
> >
> > Thank you very much.
> >
> > [1]. https://issues.apache.org/jira/browse/HBASE-21879
> > [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
> > [3]. https://issues.apache.org/jira/browse/HDFS-3246
> >
> >
>
Re: Please help to review patch of the HDFS-14585 (backport Byte
Posted by OpenInx <op...@gmail.com>.
> What are your challenges upgrading to Hadoop 3 ?
It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams
will need some time to test the stability
and performance, so I think the HBase will not upgrade to hadoop 3 so quick
in short time. Backporting the ByteBuffer
pread to hadoop 2.9 is more suitable for now (it's a minor feature, should
be OK to port in branch-2.9). I believe in future
HBase release we will include the hadoop 3
> Did you verify if your problem statement valid, if you use Ozone API ?
We have some work about the object-store api [1]. but I'm not quite sure
how will it affect the GC. will take a look about that.
Thanks for your reply.
[1]. https://issues.apache.org/jira/browse/HBASE-22149
On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sr...@yahoo.com> wrote:
> What are your challenges upgrading to Hadoop 3 ?
>
> Did you verify if your problem statement valid, if you use Ozone API ?
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Thu, Jun 27, 2019 at 9:20 AM, OpenInx
> <op...@gmail.com> wrote:
> Dear HDFS Team:
>
> I'm apache HBase member and working on HBASE-21879 [1]: it's a feature
> which help to optimize the
> HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
> HDFS block into offheap
> ByteBuffers and it had great performance improvement in our benchmark test
> [2]. While the offheap
> HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
> as: ByteBuffer stream
> read and ByteBuffer positional read.
>
> The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3]
> for Hadoop 3.3.0 version. but
> the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
> upgrade the dependency to
> Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 so
> that in the next hadoop
> 2.9.3 can include it , and our Apache HBase can upgrade the dependency to
> 2.9.3.
>
> Now all the patches of HBASE-21879 have been merged into HBase master
> branch. If we cannot upgrade to
> the Hadoop which include the ByteBuffer pread implementation, then I
> believe almost all of work from HBASE-21879
> will worth nothing because we still read heap block and still high GC
> pressure in some special case ( such as
> cacheHitRatio is not very high). That will be quite frustrating for us.
>
> Actually, we have been pushing the backport for the past few weeks. our
> XiaoMi HDFS team have prepared the
> backport patches, but seems the guys we ping didn't have the bandwidth to
> help review the patches. we
> expect the backport can be included in Hadoop 2.9.3 so that our upstream
> can benefit from this as soon as
> possible.
>
> If any HDFS committer or PMC have some bandwidth to review the backport
> patches, please give us a hand.
> The Apache HBase team really need your help.
>
> The ByteBuffer pread backport issues are:
> https://issues.apache.org/jira/browse/HDFS-14585
> https://issues.apache.org/jira/browse/HDFS-14483
>
> Thank you very much.
>
> [1]. https://issues.apache.org/jira/browse/HBASE-21879
> [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
> [3]. https://issues.apache.org/jira/browse/HDFS-3246
>
>
Re: Please help to review patch of the HDFS-14585 (backport Byte
Posted by OpenInx <op...@gmail.com>.
> What are your challenges upgrading to Hadoop 3 ?
It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams
will need some time to test the stability
and performance, so I think the HBase will not upgrade to hadoop 3 so quick
in short time. Backporting the ByteBuffer
pread to hadoop 2.9 is more suitable for now (it's a minor feature, should
be OK to port in branch-2.9). I believe in future
HBase release we will include the hadoop 3
> Did you verify if your problem statement valid, if you use Ozone API ?
We have some work about the object-store api [1]. but I'm not quite sure
how will it affect the GC. will take a look about that.
Thanks for your reply.
[1]. https://issues.apache.org/jira/browse/HBASE-22149
On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sr...@yahoo.com> wrote:
> What are your challenges upgrading to Hadoop 3 ?
>
> Did you verify if your problem statement valid, if you use Ozone API ?
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Thu, Jun 27, 2019 at 9:20 AM, OpenInx
> <op...@gmail.com> wrote:
> Dear HDFS Team:
>
> I'm apache HBase member and working on HBASE-21879 [1]: it's a feature
> which help to optimize the
> HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
> HDFS block into offheap
> ByteBuffers and it had great performance improvement in our benchmark test
> [2]. While the offheap
> HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
> as: ByteBuffer stream
> read and ByteBuffer positional read.
>
> The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3]
> for Hadoop 3.3.0 version. but
> the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
> upgrade the dependency to
> Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 so
> that in the next hadoop
> 2.9.3 can include it , and our Apache HBase can upgrade the dependency to
> 2.9.3.
>
> Now all the patches of HBASE-21879 have been merged into HBase master
> branch. If we cannot upgrade to
> the Hadoop which include the ByteBuffer pread implementation, then I
> believe almost all of work from HBASE-21879
> will worth nothing because we still read heap block and still high GC
> pressure in some special case ( such as
> cacheHitRatio is not very high). That will be quite frustrating for us.
>
> Actually, we have been pushing the backport for the past few weeks. our
> XiaoMi HDFS team have prepared the
> backport patches, but seems the guys we ping didn't have the bandwidth to
> help review the patches. we
> expect the backport can be included in Hadoop 2.9.3 so that our upstream
> can benefit from this as soon as
> possible.
>
> If any HDFS committer or PMC have some bandwidth to review the backport
> patches, please give us a hand.
> The Apache HBase team really need your help.
>
> The ByteBuffer pread backport issues are:
> https://issues.apache.org/jira/browse/HDFS-14585
> https://issues.apache.org/jira/browse/HDFS-14483
>
> Thank you very much.
>
> [1]. https://issues.apache.org/jira/browse/HBASE-21879
> [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
> [3]. https://issues.apache.org/jira/browse/HDFS-3246
>
>