You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gluten.apache.org by Terry Wang <zj...@gmail.com> on 2024/12/03 12:16:20 UTC

RFC [VL] Move ColumnarBuildSideRelation's memory occupation to Spark off-heap

Hi, everyone:

We have just proposed an improvement on optimizing the memory usage of
ColumnarBuildSideRelation (https://docs.goo
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
gle
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
.com/
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
docu
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
ment
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
/d/1
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
eZNW
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
PUE
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
diz2
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
JPJf
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
hy
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
Vn9
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
hrk6
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
SqJ
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
FRNz
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
OMZ
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
m6u
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
5Yr
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
edk
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
/edi
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
t
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
?usp
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
=sh
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
ari
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
ng
<https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>)

and opened a draft merge request (https://github.com/
<https://github.com/apache/incubator-gluten/pull/8127>apac
<https://github.com/apache/incubator-gluten/pull/8127>he/
<https://github.com/apache/incubator-gluten/pull/8127>inc
<https://github.com/apache/incubator-gluten/pull/8127>ubat
<https://github.com/apache/incubator-gluten/pull/8127>or-
<https://github.com/apache/incubator-gluten/pull/8127>gl
<https://github.com/apache/incubator-gluten/pull/8127>ute
<https://github.com/apache/incubator-gluten/pull/8127>n/p
<https://github.com/apache/incubator-gluten/pull/8127>ull
<https://github.com/apache/incubator-gluten/pull/8127>/81
<https://github.com/apache/incubator-gluten/pull/8127>27
<https://github.com/apache/incubator-gluten/pull/8127>) to implement this
idea.

We are very eager to hear your thoughts and opinions on this improvement.
Are there any potential issues or concerns that you can foresee? We believe
that your input will be invaluable in making this optimization as effective
as possible.

--
Best Regards,
Terry Wang

Re: RFC [VL] Move ColumnarBuildSideRelation's memory occupation to Spark off-heap

Posted by Terry Wang <zj...@gmail.com>.

Hi folks~
I add more ut and verify the patch through tpcds in my own dev box.
Please help review https://github.com/apache/incubator-gluten/pull/8127
when your guys have time!
Thanks


On Thu, Dec 5, 2024 at 1:20 PM Hongze Zhang <ho...@apache.org> wrote:

> Thank you for the effort! I see your PR
> https://github.com/apache/incubator-gluten/pull/8127 was filed. I will
> take an initial look and please kindly let the community know once
> it's ready for review.
>
> Hongze
>
> On Tue, Dec 3, 2024 at 8:16 PM Terry Wang <zj...@gmail.com> wrote:
> >
> > Hi, everyone:
> >
> > We have just proposed an improvement on optimizing the memory usage of
> > ColumnarBuildSideRelation (https://docs.goo
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > gle
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > .com/
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > docu
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > ment
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > /d/1
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > eZNW
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > PUE
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > diz2
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > JPJf
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > hy
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > Vn9
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > hrk6
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > SqJ
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > FRNz
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > OMZ
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > m6u
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > 5Yr
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > edk
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > /edi
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > t
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > ?usp
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > =sh
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > ari
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >
> > ng
> > <
> https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing
> >)
> >
> > and opened a draft merge request (https://github.com/
> > <https://github.com/apache/incubator-gluten/pull/8127>apac
> > <https://github.com/apache/incubator-gluten/pull/8127>he/
> > <https://github.com/apache/incubator-gluten/pull/8127>inc
> > <https://github.com/apache/incubator-gluten/pull/8127>ubat
> > <https://github.com/apache/incubator-gluten/pull/8127>or-
> > <https://github.com/apache/incubator-gluten/pull/8127>gl
> > <https://github.com/apache/incubator-gluten/pull/8127>ute
> > <https://github.com/apache/incubator-gluten/pull/8127>n/p
> > <https://github.com/apache/incubator-gluten/pull/8127>ull
> > <https://github.com/apache/incubator-gluten/pull/8127>/81
> > <https://github.com/apache/incubator-gluten/pull/8127>27
> > <https://github.com/apache/incubator-gluten/pull/8127>) to implement
> this
> > idea.
> >
> > We are very eager to hear your thoughts and opinions on this improvement.
> > Are there any potential issues or concerns that you can foresee? We
> believe
> > that your input will be invaluable in making this optimization as
> effective
> > as possible.
> >
> > --
> > Best Regards,
> > Terry Wang
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@gluten.apache.org
> For additional commands, e-mail: dev-help@gluten.apache.org
>
>

-- 
Best Regards,
Terry Wang

Re: RFC [VL] Move ColumnarBuildSideRelation's memory occupation to Spark off-heap

Posted by Hongze Zhang <ho...@apache.org>.

Thank you for the effort! I see your PR
https://github.com/apache/incubator-gluten/pull/8127 was filed. I will
take an initial look and please kindly let the community know once
it's ready for review.

Hongze

On Tue, Dec 3, 2024 at 8:16 PM Terry Wang <zj...@gmail.com> wrote:
>
> Hi, everyone:
>
> We have just proposed an improvement on optimizing the memory usage of
> ColumnarBuildSideRelation (https://docs.goo
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> gle
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> .com/
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> docu
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> ment
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> /d/1
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> eZNW
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> PUE
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> diz2
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> JPJf
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> hy
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> Vn9
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> hrk6
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> SqJ
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> FRNz
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> OMZ
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> m6u
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> 5Yr
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> edk
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> /edi
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> t
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> ?usp
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> =sh
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> ari
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>
> ng
> <https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?usp=sharing>)
>
> and opened a draft merge request (https://github.com/
> <https://github.com/apache/incubator-gluten/pull/8127>apac
> <https://github.com/apache/incubator-gluten/pull/8127>he/
> <https://github.com/apache/incubator-gluten/pull/8127>inc
> <https://github.com/apache/incubator-gluten/pull/8127>ubat
> <https://github.com/apache/incubator-gluten/pull/8127>or-
> <https://github.com/apache/incubator-gluten/pull/8127>gl
> <https://github.com/apache/incubator-gluten/pull/8127>ute
> <https://github.com/apache/incubator-gluten/pull/8127>n/p
> <https://github.com/apache/incubator-gluten/pull/8127>ull
> <https://github.com/apache/incubator-gluten/pull/8127>/81
> <https://github.com/apache/incubator-gluten/pull/8127>27
> <https://github.com/apache/incubator-gluten/pull/8127>) to implement this
> idea.
>
> We are very eager to hear your thoughts and opinions on this improvement.
> Are there any potential issues or concerns that you can foresee? We believe
> that your input will be invaluable in making this optimization as effective
> as possible.
>
> --
> Best Regards,
> Terry Wang

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@gluten.apache.org
For additional commands, e-mail: dev-help@gluten.apache.org