You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by yangzhg <ya...@apache.org> on 2022/04/12 11:28:57 UTC
[Discussion] Remove AlphaRowset Related Code
HI all:
BetaRowset has been online for a long time, and has been the default
format, performance and stability have been fully verified, currently in
our code base still retains support for AlphaRowset, while taking into
account the support of AlphaRowset for our follow-up Development has added
a lot of unnecessary work, so I propose to remove the alphaRowset support
and code in doris.
This work needs to be done in two steps.
1. add the stmt to detect alpharowset tables in version 1.1, before 1.1
upgrade to 1.2 users need to convert all tables in the cluster to
betarowset, and we also can add a configuration to control whether to
automatically compaction alpha rowset tables to betarowset format
2. remove alpha rowset related code in 1.2 version
Re: [Discussion] Remove AlphaRowset Related Code
Posted by 王博 <wa...@gmail.com>.
It seems that we still need to check whether all rowsets are converted to
v2.
Relying on compaction cannot guarantee that all rowsets can complete
automatic conversion operations
zhg yang <ya...@gmail.com> 于2022年4月13日周三 13:27写道：
> 1. there are two ways to rigger convert:
> a. manually: user can trigger manually by execute 'ALTER TABLE
> table_name SET ("storage_format" = "v2");'
> b. automatic: I will add a config like 'compaction_v1_to_v2' if this is
> true, base compaction will auto compaction v1 tablet to v2 even there is no
> data load.
> c. verify: I will add a stmt like "show storage format from database"
> to check if all table is v2
> 2. the cost of conversion is the same as directly schema change, user can
> select manually or automatic way to apply the conversion according to
> their situation
>
> Thanks
> Yang Zhengguo
>
>
> 王博 <wa...@gmail.com> 于2022年4月13日周三 12:03写道：
>
> > I don't doubt the need to remove the v1 code,My main focus is to *keep
> > Doris stable* while removing the v1 code.
> > 1 First, How to trigger the convert, can you briefly describe the main
> > process? Is it automatic or requires manual operation?
> > 2 Second, what is the cost of converting(will it take up too much cpu and
> > disk IO?), especially when a Doris cluster has tens of millions tablets.
> >
> > yangzhg <ya...@apache.org> 于2022年4月12日周二 19:29写道：
> >
> > > HI all:
> > > BetaRowset has been online for a long time, and has been the default
> > > format, performance and stability have been fully verified, currently
> in
> > > our code base still retains support for AlphaRowset, while taking into
> > > account the support of AlphaRowset for our follow-up Development has
> > added
> > > a lot of unnecessary work, so I propose to remove the alphaRowset
> support
> > > and code in doris.
> > >
> > > This work needs to be done in two steps.
> > > 1. add the stmt to detect alpharowset tables in version 1.1, before 1.1
> > > upgrade to 1.2 users need to convert all tables in the cluster to
> > > betarowset, and we also can add a configuration to control whether to
> > > automatically compaction alpha rowset tables to betarowset format
> > > 2. remove alpha rowset related code in 1.2 version
> > >
> >
> >
> > --
> > 王博 Wang Bo
> >
>
--
王博 Wang Bo
Re: [Discussion] Remove AlphaRowset Related Code
Posted by zhg yang <ya...@gmail.com>.
1. there are two ways to rigger convert:
a. manually: user can trigger manually by execute 'ALTER TABLE
table_name SET ("storage_format" = "v2");'
b. automatic: I will add a config like 'compaction_v1_to_v2' if this is
true, base compaction will auto compaction v1 tablet to v2 even there is no
data load.
c. verify: I will add a stmt like "show storage format from database"
to check if all table is v2
2. the cost of conversion is the same as directly schema change, user can
select manually or automatic way to apply the conversion according to
their situation
Thanks
Yang Zhengguo
王博 <wa...@gmail.com> 于2022年4月13日周三 12:03写道：
> I don't doubt the need to remove the v1 code,My main focus is to *keep
> Doris stable* while removing the v1 code.
> 1 First, How to trigger the convert, can you briefly describe the main
> process? Is it automatic or requires manual operation?
> 2 Second, what is the cost of converting(will it take up too much cpu and
> disk IO?), especially when a Doris cluster has tens of millions tablets.
>
> yangzhg <ya...@apache.org> 于2022年4月12日周二 19:29写道：
>
> > HI all:
> > BetaRowset has been online for a long time, and has been the default
> > format, performance and stability have been fully verified, currently in
> > our code base still retains support for AlphaRowset, while taking into
> > account the support of AlphaRowset for our follow-up Development has
> added
> > a lot of unnecessary work, so I propose to remove the alphaRowset support
> > and code in doris.
> >
> > This work needs to be done in two steps.
> > 1. add the stmt to detect alpharowset tables in version 1.1, before 1.1
> > upgrade to 1.2 users need to convert all tables in the cluster to
> > betarowset, and we also can add a configuration to control whether to
> > automatically compaction alpha rowset tables to betarowset format
> > 2. remove alpha rowset related code in 1.2 version
> >
>
>
> --
> 王博 Wang Bo
>
Re: [Discussion] Remove AlphaRowset Related Code
Posted by 王博 <wa...@gmail.com>.
I don't doubt the need to remove the v1 code,My main focus is to *keep
Doris stable* while removing the v1 code.
1 First, How to trigger the convert, can you briefly describe the main
process? Is it automatic or requires manual operation?
2 Second, what is the cost of converting(will it take up too much cpu and
disk IO?), especially when a Doris cluster has tens of millions tablets.
yangzhg <ya...@apache.org> 于2022年4月12日周二 19:29写道：
> HI all:
> BetaRowset has been online for a long time, and has been the default
> format, performance and stability have been fully verified, currently in
> our code base still retains support for AlphaRowset, while taking into
> account the support of AlphaRowset for our follow-up Development has added
> a lot of unnecessary work, so I propose to remove the alphaRowset support
> and code in doris.
>
> This work needs to be done in two steps.
> 1. add the stmt to detect alpharowset tables in version 1.1, before 1.1
> upgrade to 1.2 users need to convert all tables in the cluster to
> betarowset, and we also can add a configuration to control whether to
> automatically compaction alpha rowset tables to betarowset format
> 2. remove alpha rowset related code in 1.2 version
>
--
王博 Wang Bo
Re:[Discussion] Remove AlphaRowset Related Code
Posted by 陈明雨 <mo...@163.com>.
Indeed, we need to remove it.
And we need to command or tool to make sure that all rowset can be converted to beta rowset, controllably.
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
At 2022-04-12 19:28:57, "yangzhg" <ya...@apache.org> wrote:
>HI all:
>BetaRowset has been online for a long time, and has been the default
>format, performance and stability have been fully verified, currently in
>our code base still retains support for AlphaRowset, while taking into
>account the support of AlphaRowset for our follow-up Development has added
>a lot of unnecessary work, so I propose to remove the alphaRowset support
>and code in doris.
>
>This work needs to be done in two steps.
>1. add the stmt to detect alpharowset tables in version 1.1, before 1.1
>upgrade to 1.2 users need to convert all tables in the cluster to
>betarowset, and we also can add a configuration to control whether to
>automatically compaction alpha rowset tables to betarowset format
>2. remove alpha rowset related code in 1.2 version