You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Xintong Song <to...@gmail.com> on 2020/09/03 10:42:34 UTC
[VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Hi devs,
I'd like to start a voting thread on FLIP-141[1], which proposes how
managed memory should be shared by various use cases within a slot. The
proposal has been discussed in [2].
The vote will be open for at least 72h + weekends. I'll try to close it on
September 8, unless there is an objection or not enough votes.
Thank you~
Xintong Song
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Xintong Song <to...@gmail.com>.
Thanks everyone,
I'm closing this vote now in a separate email.
Concerning the naming, I will use DATAPROC, as @Stephan suggested in the
discussion thread [1], for now. If there are any other opinions, feel free
to reach out to me anytime before the release.
Thank you~
Xintong Song
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44533.html
On Wed, Sep 9, 2020 at 4:02 PM Andrey Zagrebin <az...@apache.org> wrote:
> For the option name, maybe:
> *flink.main*
> or
> *flink.managed* (this may be a bit confusing for existing users as we said
> that the overall managed memory is managed by Flink)
>
> On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin <az...@apache.org>
> wrote:
>
> > +1
> >
> > Best,
> > Andrey
> >
> > On Tue, Sep 8, 2020 at 2:16 PM Yu Li <ca...@gmail.com> wrote:
> >
> >> +1
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <al...@apache.org>
> >> wrote:
> >>
> >> > +1
> >> >
> >> > We just need to make sure to find a good name before the release but
> >> > shouldn't block any work on this.
> >> >
> >> > Aljoscha
> >> >
> >> > On 08.09.20 07:59, Xintong Song wrote:
> >> > > Thanks for the vote, @Jincheng.
> >> > >
> >> > >
> >> > > Concerning the namings, the original idea was, as you suggested, to
> >> have
> >> > > separate configuration names for batch and rocksdb while only one of
> >> them
> >> > > will take effect at a time.
> >> > >
> >> > >
> >> > > It was then in the discussion thread [1] that @Stepahn suggested to
> >> > combine
> >> > > these two.
> >> > >
> >> > >> We never have batch algos and RocksDB mixed, having this as
> >> > separate
> >> > >> options is confusing as it suggests this can be combined
> >> arbitrarily. I
> >> > >> also think that a slim possibility that we may ever combine this in
> >> the
> >> > >> future is not enough reason to make it more complex/confusing.
> >> > >>
> >> > >
> >> > > This suggestion was also supported by others in the discussion
> thread.
> >> > > That's why we are trying to come up with a name that covers both
> batch
> >> > and
> >> > > rocksdb memory consumers.
> >> > >
> >> > >
> >> > > Thank you~
> >> > >
> >> > > Xintong Song
> >> > >
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> >> > >
> >> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <
> sunjincheng121@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > >> +1 for the proposal!
> >> > >>
> >> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> >> > configuration
> >> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> >> > "ROCKSDB"
> >> > >> for roockdb. and the default value as follows:
> >> > >>
> >> > >> {
> >> > >> BATCH_OP: 70,
> >> > >> ROCKSDB : 70,
> >> > >> PYTHON : 30
> >> > >> }
> >> > >>
> >> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> >> > >>
> >> > >> Best,
> >> > >> Jincheng
> >> > >>
> >> > >>
> >> > >> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
> >> > >>
> >> > >>> Thanks for the votes.
> >> > >>>
> >> > >>> Concerning the name for batch/RocksDB memory consumer, how about
> >> > >> "execution
> >> > >>> memory"?
> >> > >>> We can further explain in docs and config option description that
> >> this
> >> > is
> >> > >>> used for job execution, which is currently dedicated to rocksdb in
> >> > >>> streaming and batch algorithms in batch.
> >> > >>>
> >> > >>> Thank you~
> >> > >>>
> >> > >>> Xintong Song
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com>
> >> wrote:
> >> > >>>
> >> > >>>> +1
> >> > >>>>
> >> > >>>> Best,
> >> > >>>> Yangze Guo
> >> > >>>>
> >> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com>
> wrote:
> >> > >>>>>
> >> > >>>>> +1
> >> > >>>>>
> >> > >>>>> Thanks,
> >> > >>>>> Zhu
> >> > >>>>>
> >> > >>>>> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> >> > >>>>>
> >> > >>>>>> +1
> >> > >>>>>>
> >> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> >> > >>>>>>>
> >> > >>>>>>> Hi Xintong,
> >> > >>>>>>>
> >> > >>>>>>> thanks for starting the vote.
> >> > >>>>>>>
> >> > >>>>>>> +1 for the proposal given that we find a proper name for the
> >> > >>>>>>> different memory consumers (specifically the batch/RocksDB
> >> > >>> consumer)
> >> > >>>> and
> >> > >>>>>>> their corresponding weights.
> >> > >>>>>>>
> >> > >>>>>>> Cheers,
> >> > >>>>>>> Till
> >> > >>>>>>>
> >> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> >> > >>> tonysong820@gmail.com>
> >> > >>>>>> wrote:
> >> > >>>>>>>
> >> > >>>>>>>> Hi devs,
> >> > >>>>>>>>
> >> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which
> >> proposes
> >> > >>> how
> >> > >>>>>>>> managed memory should be shared by various use cases within a
> >> > >>> slot.
> >> > >>>> The
> >> > >>>>>>>> proposal has been discussed in [2].
> >> > >>>>>>>>
> >> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try
> to
> >> > >>>> close it
> >> > >>>>>> on
> >> > >>>>>>>> September 8, unless there is an objection or not enough
> votes.
> >> > >>>>>>>>
> >> > >>>>>>>> Thank you~
> >> > >>>>>>>>
> >> > >>>>>>>> Xintong Song
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>> [1]
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >> > >>>>>>>>
> >> > >>>>>>>> [2]
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> > >
> >> >
> >> >
> >>
> >
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Andrey Zagrebin <az...@apache.org>.
For the option name, maybe:
*flink.main*
or
*flink.managed* (this may be a bit confusing for existing users as we said
that the overall managed memory is managed by Flink)
On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin <az...@apache.org> wrote:
> +1
>
> Best,
> Andrey
>
> On Tue, Sep 8, 2020 at 2:16 PM Yu Li <ca...@gmail.com> wrote:
>
>> +1
>>
>> Best Regards,
>> Yu
>>
>>
>> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>> > +1
>> >
>> > We just need to make sure to find a good name before the release but
>> > shouldn't block any work on this.
>> >
>> > Aljoscha
>> >
>> > On 08.09.20 07:59, Xintong Song wrote:
>> > > Thanks for the vote, @Jincheng.
>> > >
>> > >
>> > > Concerning the namings, the original idea was, as you suggested, to
>> have
>> > > separate configuration names for batch and rocksdb while only one of
>> them
>> > > will take effect at a time.
>> > >
>> > >
>> > > It was then in the discussion thread [1] that @Stepahn suggested to
>> > combine
>> > > these two.
>> > >
>> > >> We never have batch algos and RocksDB mixed, having this as
>> > separate
>> > >> options is confusing as it suggests this can be combined
>> arbitrarily. I
>> > >> also think that a slim possibility that we may ever combine this in
>> the
>> > >> future is not enough reason to make it more complex/confusing.
>> > >>
>> > >
>> > > This suggestion was also supported by others in the discussion thread.
>> > > That's why we are trying to come up with a name that covers both batch
>> > and
>> > > rocksdb memory consumers.
>> > >
>> > >
>> > > Thank you~
>> > >
>> > > Xintong Song
>> > >
>> > >
>> > > [1]
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
>> > >
>> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <sunjincheng121@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> +1 for the proposal!
>> > >>
>> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
>> > configuration
>> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
>> > "ROCKSDB"
>> > >> for roockdb. and the default value as follows:
>> > >>
>> > >> {
>> > >> BATCH_OP: 70,
>> > >> ROCKSDB : 70,
>> > >> PYTHON : 30
>> > >> }
>> > >>
>> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>> > >>
>> > >> Best,
>> > >> Jincheng
>> > >>
>> > >>
>> > >> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
>> > >>
>> > >>> Thanks for the votes.
>> > >>>
>> > >>> Concerning the name for batch/RocksDB memory consumer, how about
>> > >> "execution
>> > >>> memory"?
>> > >>> We can further explain in docs and config option description that
>> this
>> > is
>> > >>> used for job execution, which is currently dedicated to rocksdb in
>> > >>> streaming and batch algorithms in batch.
>> > >>>
>> > >>> Thank you~
>> > >>>
>> > >>> Xintong Song
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com>
>> wrote:
>> > >>>
>> > >>>> +1
>> > >>>>
>> > >>>> Best,
>> > >>>> Yangze Guo
>> > >>>>
>> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
>> > >>>>>
>> > >>>>> +1
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Zhu
>> > >>>>>
>> > >>>>> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
>> > >>>>>
>> > >>>>>> +1
>> > >>>>>>
>> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
>> > >>>>>>>
>> > >>>>>>> Hi Xintong,
>> > >>>>>>>
>> > >>>>>>> thanks for starting the vote.
>> > >>>>>>>
>> > >>>>>>> +1 for the proposal given that we find a proper name for the
>> > >>>>>>> different memory consumers (specifically the batch/RocksDB
>> > >>> consumer)
>> > >>>> and
>> > >>>>>>> their corresponding weights.
>> > >>>>>>>
>> > >>>>>>> Cheers,
>> > >>>>>>> Till
>> > >>>>>>>
>> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
>> > >>> tonysong820@gmail.com>
>> > >>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>>> Hi devs,
>> > >>>>>>>>
>> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which
>> proposes
>> > >>> how
>> > >>>>>>>> managed memory should be shared by various use cases within a
>> > >>> slot.
>> > >>>> The
>> > >>>>>>>> proposal has been discussed in [2].
>> > >>>>>>>>
>> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
>> > >>>> close it
>> > >>>>>> on
>> > >>>>>>>> September 8, unless there is an objection or not enough votes.
>> > >>>>>>>>
>> > >>>>>>>> Thank you~
>> > >>>>>>>>
>> > >>>>>>>> Xintong Song
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> [1]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>> > >>>>>>>>
>> > >>>>>>>> [2]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> >
>> >
>>
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Andrey Zagrebin <az...@apache.org>.
+1
Best,
Andrey
On Tue, Sep 8, 2020 at 2:16 PM Yu Li <ca...@gmail.com> wrote:
> +1
>
> Best Regards,
> Yu
>
>
> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <al...@apache.org> wrote:
>
> > +1
> >
> > We just need to make sure to find a good name before the release but
> > shouldn't block any work on this.
> >
> > Aljoscha
> >
> > On 08.09.20 07:59, Xintong Song wrote:
> > > Thanks for the vote, @Jincheng.
> > >
> > >
> > > Concerning the namings, the original idea was, as you suggested, to
> have
> > > separate configuration names for batch and rocksdb while only one of
> them
> > > will take effect at a time.
> > >
> > >
> > > It was then in the discussion thread [1] that @Stepahn suggested to
> > combine
> > > these two.
> > >
> > >> We never have batch algos and RocksDB mixed, having this as
> > separate
> > >> options is confusing as it suggests this can be combined arbitrarily.
> I
> > >> also think that a slim possibility that we may ever combine this in
> the
> > >> future is not enough reason to make it more complex/confusing.
> > >>
> > >
> > > This suggestion was also supported by others in the discussion thread.
> > > That's why we are trying to come up with a name that covers both batch
> > and
> > > rocksdb memory consumers.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> > >
> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <su...@gmail.com>
> > > wrote:
> > >
> > >> +1 for the proposal!
> > >>
> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> > configuration
> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> > "ROCKSDB"
> > >> for roockdb. and the default value as follows:
> > >>
> > >> {
> > >> BATCH_OP: 70,
> > >> ROCKSDB : 70,
> > >> PYTHON : 30
> > >> }
> > >>
> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> > >>
> > >> Best,
> > >> Jincheng
> > >>
> > >>
> > >> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
> > >>
> > >>> Thanks for the votes.
> > >>>
> > >>> Concerning the name for batch/RocksDB memory consumer, how about
> > >> "execution
> > >>> memory"?
> > >>> We can further explain in docs and config option description that
> this
> > is
> > >>> used for job execution, which is currently dedicated to rocksdb in
> > >>> streaming and batch algorithms in batch.
> > >>>
> > >>> Thank you~
> > >>>
> > >>> Xintong Song
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com>
> wrote:
> > >>>
> > >>>> +1
> > >>>>
> > >>>> Best,
> > >>>> Yangze Guo
> > >>>>
> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
> > >>>>>
> > >>>>> +1
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Zhu
> > >>>>>
> > >>>>> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> > >>>>>>>
> > >>>>>>> Hi Xintong,
> > >>>>>>>
> > >>>>>>> thanks for starting the vote.
> > >>>>>>>
> > >>>>>>> +1 for the proposal given that we find a proper name for the
> > >>>>>>> different memory consumers (specifically the batch/RocksDB
> > >>> consumer)
> > >>>> and
> > >>>>>>> their corresponding weights.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> > >>> tonysong820@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi devs,
> > >>>>>>>>
> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
> > >>> how
> > >>>>>>>> managed memory should be shared by various use cases within a
> > >>> slot.
> > >>>> The
> > >>>>>>>> proposal has been discussed in [2].
> > >>>>>>>>
> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
> > >>>> close it
> > >>>>>> on
> > >>>>>>>> September 8, unless there is an objection or not enough votes.
> > >>>>>>>>
> > >>>>>>>> Thank you~
> > >>>>>>>>
> > >>>>>>>> Xintong Song
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > >>>>>>>>
> > >>>>>>>> [2]
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Yu Li <ca...@gmail.com>.
+1
Best Regards,
Yu
On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <al...@apache.org> wrote:
> +1
>
> We just need to make sure to find a good name before the release but
> shouldn't block any work on this.
>
> Aljoscha
>
> On 08.09.20 07:59, Xintong Song wrote:
> > Thanks for the vote, @Jincheng.
> >
> >
> > Concerning the namings, the original idea was, as you suggested, to have
> > separate configuration names for batch and rocksdb while only one of them
> > will take effect at a time.
> >
> >
> > It was then in the discussion thread [1] that @Stepahn suggested to
> combine
> > these two.
> >
> >> We never have batch algos and RocksDB mixed, having this as
> separate
> >> options is confusing as it suggests this can be combined arbitrarily. I
> >> also think that a slim possibility that we may ever combine this in the
> >> future is not enough reason to make it more complex/confusing.
> >>
> >
> > This suggestion was also supported by others in the discussion thread.
> > That's why we are trying to come up with a name that covers both batch
> and
> > rocksdb memory consumers.
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> >
> > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <su...@gmail.com>
> > wrote:
> >
> >> +1 for the proposal!
> >>
> >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> configuration
> >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> "ROCKSDB"
> >> for roockdb. and the default value as follows:
> >>
> >> {
> >> BATCH_OP: 70,
> >> ROCKSDB : 70,
> >> PYTHON : 30
> >> }
> >>
> >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
> >>
> >>> Thanks for the votes.
> >>>
> >>> Concerning the name for batch/RocksDB memory consumer, how about
> >> "execution
> >>> memory"?
> >>> We can further explain in docs and config option description that this
> is
> >>> used for job execution, which is currently dedicated to rocksdb in
> >>> streaming and batch algorithms in batch.
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>>
> >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Best,
> >>>> Yangze Guo
> >>>>
> >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
> >>>>>
> >>>>> +1
> >>>>>
> >>>>> Thanks,
> >>>>> Zhu
> >>>>>
> >>>>> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> >>>>>>>
> >>>>>>> Hi Xintong,
> >>>>>>>
> >>>>>>> thanks for starting the vote.
> >>>>>>>
> >>>>>>> +1 for the proposal given that we find a proper name for the
> >>>>>>> different memory consumers (specifically the batch/RocksDB
> >>> consumer)
> >>>> and
> >>>>>>> their corresponding weights.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>>
> >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> >>> tonysong820@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi devs,
> >>>>>>>>
> >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
> >>> how
> >>>>>>>> managed memory should be shared by various use cases within a
> >>> slot.
> >>>> The
> >>>>>>>> proposal has been discussed in [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
> >>>> close it
> >>>>>> on
> >>>>>>>> September 8, unless there is an objection or not enough votes.
> >>>>>>>>
> >>>>>>>> Thank you~
> >>>>>>>>
> >>>>>>>> Xintong Song
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >>>>>>>>
> >>>>>>>> [2]
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Aljoscha Krettek <al...@apache.org>.
+1
We just need to make sure to find a good name before the release but
shouldn't block any work on this.
Aljoscha
On 08.09.20 07:59, Xintong Song wrote:
> Thanks for the vote, @Jincheng.
>
>
> Concerning the namings, the original idea was, as you suggested, to have
> separate configuration names for batch and rocksdb while only one of them
> will take effect at a time.
>
>
> It was then in the discussion thread [1] that @Stepahn suggested to combine
> these two.
>
>> We never have batch algos and RocksDB mixed, having this as separate
>> options is confusing as it suggests this can be combined arbitrarily. I
>> also think that a slim possibility that we may ever combine this in the
>> future is not enough reason to make it more complex/confusing.
>>
>
> This suggestion was also supported by others in the discussion thread.
> That's why we are trying to come up with a name that covers both batch and
> rocksdb memory consumers.
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
>
> On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <su...@gmail.com>
> wrote:
>
>> +1 for the proposal!
>>
>> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
>> names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
>> for roockdb. and the default value as follows:
>>
>> {
>> BATCH_OP: 70,
>> ROCKSDB : 70,
>> PYTHON : 30
>> }
>>
>> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>>
>> Best,
>> Jincheng
>>
>>
>> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
>>
>>> Thanks for the votes.
>>>
>>> Concerning the name for batch/RocksDB memory consumer, how about
>> "execution
>>> memory"?
>>> We can further explain in docs and config option description that this is
>>> used for job execution, which is currently dedicated to rocksdb in
>>> streaming and batch algorithms in batch.
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> Best,
>>>> Yangze Guo
>>>>
>>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>> Thanks,
>>>>> Zhu
>>>>>
>>>>> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
>>>>>
>>>>>> +1
>>>>>>
>>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
>>>>>>>
>>>>>>> Hi Xintong,
>>>>>>>
>>>>>>> thanks for starting the vote.
>>>>>>>
>>>>>>> +1 for the proposal given that we find a proper name for the
>>>>>>> different memory consumers (specifically the batch/RocksDB
>>> consumer)
>>>> and
>>>>>>> their corresponding weights.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
>>> tonysong820@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi devs,
>>>>>>>>
>>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
>>> how
>>>>>>>> managed memory should be shared by various use cases within a
>>> slot.
>>>> The
>>>>>>>> proposal has been discussed in [2].
>>>>>>>>
>>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
>>>> close it
>>>>>> on
>>>>>>>> September 8, unless there is an objection or not enough votes.
>>>>>>>>
>>>>>>>> Thank you~
>>>>>>>>
>>>>>>>> Xintong Song
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>>>>>>>>
>>>>>>>> [2]
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Xintong Song <to...@gmail.com>.
Thanks for the vote, @Jincheng.
Concerning the namings, the original idea was, as you suggested, to have
separate configuration names for batch and rocksdb while only one of them
will take effect at a time.
It was then in the discussion thread [1] that @Stepahn suggested to combine
these two.
> We never have batch algos and RocksDB mixed, having this as separate
> options is confusing as it suggests this can be combined arbitrarily. I
> also think that a slim possibility that we may ever combine this in the
> future is not enough reason to make it more complex/confusing.
>
This suggestion was also supported by others in the discussion thread.
That's why we are trying to come up with a name that covers both batch and
rocksdb memory consumers.
Thank you~
Xintong Song
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <su...@gmail.com>
wrote:
> +1 for the proposal!
>
> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
> names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
> for roockdb. and the default value as follows:
>
> {
> BATCH_OP: 70,
> ROCKSDB : 70,
> PYTHON : 30
> }
>
> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>
> Best,
> Jincheng
>
>
> Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
>
> > Thanks for the votes.
> >
> > Concerning the name for batch/RocksDB memory consumer, how about
> "execution
> > memory"?
> > We can further explain in docs and config option description that this is
> > used for job execution, which is currently dedicated to rocksdb in
> > streaming and batch algorithms in batch.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com> wrote:
> >
> > > +1
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
> > > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> > > >
> > > > > +1
> > > > >
> > > > > > 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> > > > > >
> > > > > > Hi Xintong,
> > > > > >
> > > > > > thanks for starting the vote.
> > > > > >
> > > > > > +1 for the proposal given that we find a proper name for the
> > > > > > different memory consumers (specifically the batch/RocksDB
> > consumer)
> > > and
> > > > > > their corresponding weights.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> > tonysong820@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi devs,
> > > > > >>
> > > > > >> I'd like to start a voting thread on FLIP-141[1], which proposes
> > how
> > > > > >> managed memory should be shared by various use cases within a
> > slot.
> > > The
> > > > > >> proposal has been discussed in [2].
> > > > > >>
> > > > > >> The vote will be open for at least 72h + weekends. I'll try to
> > > close it
> > > > > on
> > > > > >> September 8, unless there is an objection or not enough votes.
> > > > > >>
> > > > > >> Thank you~
> > > > > >>
> > > > > >> Xintong Song
> > > > > >>
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > > > >>
> > > > > >> [2]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > > > >>
> > > > >
> > > > >
> > >
> >
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by jincheng sun <su...@gmail.com>.
+1 for the proposal!
Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
for roockdb. and the default value as follows:
{
BATCH_OP: 70,
ROCKSDB : 70,
PYTHON : 30
}
Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
Best,
Jincheng
Xintong Song <to...@gmail.com> 于2020年9月7日周一 下午1:46写道：
> Thanks for the votes.
>
> Concerning the name for batch/RocksDB memory consumer, how about "execution
> memory"?
> We can further explain in docs and config option description that this is
> used for job execution, which is currently dedicated to rocksdb in
> streaming and batch algorithms in batch.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com> wrote:
>
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
> > >
> > > +1
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> > >
> > > > +1
> > > >
> > > > > 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> > > > >
> > > > > Hi Xintong,
> > > > >
> > > > > thanks for starting the vote.
> > > > >
> > > > > +1 for the proposal given that we find a proper name for the
> > > > > different memory consumers (specifically the batch/RocksDB
> consumer)
> > and
> > > > > their corresponding weights.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> tonysong820@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Hi devs,
> > > > >>
> > > > >> I'd like to start a voting thread on FLIP-141[1], which proposes
> how
> > > > >> managed memory should be shared by various use cases within a
> slot.
> > The
> > > > >> proposal has been discussed in [2].
> > > > >>
> > > > >> The vote will be open for at least 72h + weekends. I'll try to
> > close it
> > > > on
> > > > >> September 8, unless there is an objection or not enough votes.
> > > > >>
> > > > >> Thank you~
> > > > >>
> > > > >> Xintong Song
> > > > >>
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > > >>
> > > > >> [2]
> > > > >>
> > > > >>
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > > >>
> > > >
> > > >
> >
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Xintong Song <to...@gmail.com>.
Thanks for the votes.
Concerning the name for batch/RocksDB memory consumer, how about "execution
memory"?
We can further explain in docs and config option description that this is
used for job execution, which is currently dedicated to rocksdb in
streaming and batch algorithms in batch.
Thank you~
Xintong Song
On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <ka...@gmail.com> wrote:
> +1
>
> Best,
> Yangze Guo
>
> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
> >
> > +1
> >
> > Thanks,
> > Zhu
> >
> > Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> >
> > > +1
> > >
> > > > 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> > > >
> > > > Hi Xintong,
> > > >
> > > > thanks for starting the vote.
> > > >
> > > > +1 for the proposal given that we find a proper name for the
> > > > different memory consumers (specifically the batch/RocksDB consumer)
> and
> > > > their corresponding weights.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <to...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi devs,
> > > >>
> > > >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> > > >> managed memory should be shared by various use cases within a slot.
> The
> > > >> proposal has been discussed in [2].
> > > >>
> > > >> The vote will be open for at least 72h + weekends. I'll try to
> close it
> > > on
> > > >> September 8, unless there is an objection or not enough votes.
> > > >>
> > > >> Thank you~
> > > >>
> > > >> Xintong Song
> > > >>
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > >>
> > > >> [2]
> > > >>
> > > >>
> > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > >>
> > >
> > >
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Yangze Guo <ka...@gmail.com>.
+1
Best,
Yangze Guo
On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <re...@gmail.com> wrote:
>
> +1
>
> Thanks,
> Zhu
>
> Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
>
> > +1
> >
> > > 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> > >
> > > Hi Xintong,
> > >
> > > thanks for starting the vote.
> > >
> > > +1 for the proposal given that we find a proper name for the
> > > different memory consumers (specifically the batch/RocksDB consumer) and
> > > their corresponding weights.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > >> Hi devs,
> > >>
> > >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> > >> managed memory should be shared by various use cases within a slot. The
> > >> proposal has been discussed in [2].
> > >>
> > >> The vote will be open for at least 72h + weekends. I'll try to close it
> > on
> > >> September 8, unless there is an objection or not enough votes.
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >> [1]
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > >>
> > >> [2]
> > >>
> > >>
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > >>
> >
> >
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Zhu Zhu <re...@gmail.com>.
+1
Thanks,
Zhu
Dian Fu <di...@gmail.com> 于2020年9月7日周一 上午10:34写道：
> +1
>
> > 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
> >
> > Hi Xintong,
> >
> > thanks for starting the vote.
> >
> > +1 for the proposal given that we find a proper name for the
> > different memory consumers (specifically the batch/RocksDB consumer) and
> > their corresponding weights.
> >
> > Cheers,
> > Till
> >
> > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <to...@gmail.com>
> wrote:
> >
> >> Hi devs,
> >>
> >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> >> managed memory should be shared by various use cases within a slot. The
> >> proposal has been discussed in [2].
> >>
> >> The vote will be open for at least 72h + weekends. I'll try to close it
> on
> >> September 8, unless there is an objection or not enough votes.
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >>
> >> [2]
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >>
>
>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Dian Fu <di...@gmail.com>.
+1
> 在 2020年9月3日，下午8:46，Till Rohrmann <tr...@apache.org> 写道：
>
> Hi Xintong,
>
> thanks for starting the vote.
>
> +1 for the proposal given that we find a proper name for the
> different memory consumers (specifically the batch/RocksDB consumer) and
> their corresponding weights.
>
> Cheers,
> Till
>
> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <to...@gmail.com> wrote:
>
>> Hi devs,
>>
>> I'd like to start a voting thread on FLIP-141[1], which proposes how
>> managed memory should be shared by various use cases within a slot. The
>> proposal has been discussed in [2].
>>
>> The vote will be open for at least 72h + weekends. I'll try to close it on
>> September 8, unless there is an objection or not enough votes.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>>
>> [2]
>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>>
Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing
Posted by Till Rohrmann <tr...@apache.org>.
Hi Xintong,
thanks for starting the vote.
+1 for the proposal given that we find a proper name for the
different memory consumers (specifically the batch/RocksDB consumer) and
their corresponding weights.
Cheers,
Till
On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <to...@gmail.com> wrote:
> Hi devs,
>
> I'd like to start a voting thread on FLIP-141[1], which proposes how
> managed memory should be shared by various use cases within a slot. The
> proposal has been discussed in [2].
>
> The vote will be open for at least 72h + weekends. I'll try to close it on
> September 8, unless there is an objection or not enough votes.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>