You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by Zhao Chun <zh...@apache.org> on 2020/11/02 02:28:39 UTC
Re: Re: Re: Discussion on the support of V1 and V2 storage formats in
future versions
陈明雨 <mo...@163.com> 于2020年10月30日周五 下午9:30写道：
> So how to make sure that all tablets are being compacted and converted to
> V2?
> Some of tablet may not be triggered to the compaction.
>
>
> Maybe we can implement a thread to convert all V1 rowset to V2 in
backgroud.
And using a counter to count how many V1 rowset in one BE.
Thanks,
Zhao Chun
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> chenmingyu@apache.org
>
>
>
>
>
> 在 2020-10-29 14:05:31，"Zhao Chun" <zh...@apache.org> 写道：
> >陈明雨 <mo...@163.com> 于2020年10月29日周四 下午12:24写道：
> >
> >> > We can fiinish it when doing compaction, schema change.
> >
> >
> >
> >
> >> Currenlty we do have a approach to automatically convert v1 to v2, by
> >> setting config in be.conf.
> >> But this approach is totally uncontrollable, that we can't even know
> when
> >> it can finish.
> >>
> >> As far as I know, if we set default rowset type to beta in be.conf, the
> >new generated rowset by compaction or schema change will be segment V2
> >format.
> >And it works fine in my environment.
> >
> > And it is both time consuming and space consumimg.
> >
> >
> >Actually, I'm not catch you in this point. I think it is the same with in
> >V1 in time and space consuming.
> >
> >
> >>
> >> I think it may not be a big problem to keep the v1 format.
> >>
> >
> >Looks good to me except a lot of redundant code
> >
> >
> >> We only need to ensure that no new features are added to v1, and there
> are
> >> commands to help users actively convert data, and that's it.
> >>
> >> Actually there is method to do the convert, if user do a schema change,
> >the format will be converted to V2.
> >
> >Thanks,
> >Zhao Chun
> >
> >
> >>
> >>
> >> --
> >>
> >> 此致！Best Regards
> >> 陈明雨 Mingyu Chen
> >>
> >> Email:
> >> chenmingyu@apache.org
> >>
> >>
> >>
> >>
> >>
> >> 在 2020-10-29 11:20:43，"ling miao" <li...@apache.org> 写道：
> >> >I don’t think it’s a problem as long as it is divided into several
> parts.
> >> >But now there is a problem that the v2 version of the table is actually
> >> >very few, it is difficult for us to truly verify the stability of v2
> in a
> >> >certain version.
> >> >
> >> >Or are there any points that can guide users to trigger the conversion
> >> from
> >> >v1 to v2 by themselves?
> >> >Ling Miao
> >> >
> >> >Zhao Chun <zh...@apache.org> 于2020年10月29日周四 上午9:58写道：
> >> >
> >> >> 陈明雨 <mo...@163.com> 于2020年10月29日周四 上午9:56写道：
> >> >>
> >> >> > I agree to disable the use of V1 storage format for newly created
> >> tables
> >> >> > in version 0.14.
> >> >> >
> >> >> >
> >> >> > But "automatically convert to V2" is a dangerous and time-consuming
> >> >> > operation, we may need more discuss.
> >> >> >
> >> >> >
> >> >> We can fiinish it when doing compaction, schema change.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Zhao Chun
> >> >>
> >> >>
> >> >> >
> >> >> > --
> >> >> >
> >> >> > 此致！Best Regards
> >> >> > 陈明雨 Mingyu Chen
> >> >> >
> >> >> > Email:
> >> >> > chenmingyu@apache.org
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > At 2020-10-29 09:48:25, "apache" <ya...@apache.org> wrote:
> >> >> > >Hello everyone:
> >> >> > >At present, our V2 storage format has been developed for a long
> time,
> >> >> and
> >> >> > V2 has many advantages that V1 does not have. The coexistence of V1
> >> and
> >> >> V2
> >> >> > brings a lot of code and cluster maintenance costs.
> >> >> > >Therefore, I think the next version, that is, version 0.14 will
> >> disable
> >> >> > the use of V1 storage format for newly created tables. In version
> >> 0.16,
> >> >> the
> >> >> > storage format of V1 will be automatically converted to V2.
> Finally,
> >> the
> >> >> > support of V1 format will be completely removed in a later version.
> >> >> >
> >> >>
> >>
>
Re:Re: Re: Re: Discussion on the support of V1 and V2 storage
formats in future versions
Posted by 陈明雨 <mo...@163.com>.
> Maybe we can implement a thread to convert all V1 rowset to V2 in
>backgroud.
>And using a counter to count how many V1 rowset in one BE.
Yes, using a certain thread maybe a good way.
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
morningman@163.com;
morningman.cmy@gmail.com
在 2020-11-02 10:28:39，"Zhao Chun" <zh...@apache.org> 写道：
>陈明雨 <mo...@163.com> 于2020年10月30日周五 下午9:30写道：
>
>> So how to make sure that all tablets are being compacted and converted to
>> V2?
>> Some of tablet may not be triggered to the compaction.
>>
>>
>> Maybe we can implement a thread to convert all V1 rowset to V2 in
>backgroud.
>And using a counter to count how many V1 rowset in one BE.
>
>Thanks,
>Zhao Chun
>
>>
>>
>> --
>>
>> 此致！Best Regards
>> 陈明雨 Mingyu Chen
>>
>> Email:
>> chenmingyu@apache.org
>>
>>
>>
>>
>>
>> 在 2020-10-29 14:05:31，"Zhao Chun" <zh...@apache.org> 写道：
>> >陈明雨 <mo...@163.com> 于2020年10月29日周四 下午12:24写道：
>> >
>> >> > We can fiinish it when doing compaction, schema change.
>> >
>> >
>> >
>> >
>> >> Currenlty we do have a approach to automatically convert v1 to v2, by
>> >> setting config in be.conf.
>> >> But this approach is totally uncontrollable, that we can't even know
>> when
>> >> it can finish.
>> >>
>> >> As far as I know, if we set default rowset type to beta in be.conf, the
>> >new generated rowset by compaction or schema change will be segment V2
>> >format.
>> >And it works fine in my environment.
>> >
>> > And it is both time consuming and space consumimg.
>> >
>> >
>> >Actually, I'm not catch you in this point. I think it is the same with in
>> >V1 in time and space consuming.
>> >
>> >
>> >>
>> >> I think it may not be a big problem to keep the v1 format.
>> >>
>> >
>> >Looks good to me except a lot of redundant code
>> >
>> >
>> >> We only need to ensure that no new features are added to v1, and there
>> are
>> >> commands to help users actively convert data, and that's it.
>> >>
>> >> Actually there is method to do the convert, if user do a schema change,
>> >the format will be converted to V2.
>> >
>> >Thanks,
>> >Zhao Chun
>> >
>> >
>> >>
>> >>
>> >> --
>> >>
>> >> 此致！Best Regards
>> >> 陈明雨 Mingyu Chen
>> >>
>> >> Email:
>> >> chenmingyu@apache.org
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 在 2020-10-29 11:20:43，"ling miao" <li...@apache.org> 写道：
>> >> >I don’t think it’s a problem as long as it is divided into several
>> parts.
>> >> >But now there is a problem that the v2 version of the table is actually
>> >> >very few, it is difficult for us to truly verify the stability of v2
>> in a
>> >> >certain version.
>> >> >
>> >> >Or are there any points that can guide users to trigger the conversion
>> >> from
>> >> >v1 to v2 by themselves?
>> >> >Ling Miao
>> >> >
>> >> >Zhao Chun <zh...@apache.org> 于2020年10月29日周四 上午9:58写道：
>> >> >
>> >> >> 陈明雨 <mo...@163.com> 于2020年10月29日周四 上午9:56写道：
>> >> >>
>> >> >> > I agree to disable the use of V1 storage format for newly created
>> >> tables
>> >> >> > in version 0.14.
>> >> >> >
>> >> >> >
>> >> >> > But "automatically convert to V2" is a dangerous and time-consuming
>> >> >> > operation, we may need more discuss.
>> >> >> >
>> >> >> >
>> >> >> We can fiinish it when doing compaction, schema change.
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Zhao Chun
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > 此致！Best Regards
>> >> >> > 陈明雨 Mingyu Chen
>> >> >> >
>> >> >> > Email:
>> >> >> > chenmingyu@apache.org
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > At 2020-10-29 09:48:25, "apache" <ya...@apache.org> wrote:
>> >> >> > >Hello everyone:
>> >> >> > >At present, our V2 storage format has been developed for a long
>> time,
>> >> >> and
>> >> >> > V2 has many advantages that V1 does not have. The coexistence of V1
>> >> and
>> >> >> V2
>> >> >> > brings a lot of code and cluster maintenance costs.
>> >> >> > >Therefore, I think the next version, that is, version 0.14 will
>> >> disable
>> >> >> > the use of V1 storage format for newly created tables. In version
>> >> 0.16,
>> >> >> the
>> >> >> > storage format of V1 will be automatically converted to V2.
>> Finally,
>> >> the
>> >> >> > support of V1 format will be completely removed in a later version.
>> >> >> >
>> >> >>
>> >>
>>