You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Thomas Weise <th...@apache.org> on 2016/12/03 04:55:55 UTC
Re: Malhar release 3.6
The release is available already. For the announcement, would anyone want
to provide some information that can go on the blog page, similar to what
we had last time:
https://blogs.apache.org/apex/entry/apache_apex_malhar_3_5
Thanks,
Thomas
On Tue, Nov 22, 2016 at 2:14 PM, Thomas Weise <th...@apache.org> wrote:
> IMO we should not delay the release any longer. I will cut the RC tomorrow
> unless there is an objection.
>
> Thomas
>
>
> On Tue, Nov 22, 2016 at 2:05 PM, David Yan <da...@datatorrent.com> wrote:
>
>> Thomas,
>>
>> We need more time for the timebucket change and the purging feature, and
>> we
>> need to do benchmarking after those features have been implemented.
>> Is it possible to postpone the release? If not, we can shoot for
>> completing
>> the features for the 3.7.0 release.
>>
>> David
>>
>> On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise <th...@apache.org> wrote:
>>
>> > David, Siyuan,
>> >
>> > Any update, I would like to cut the RC.
>> >
>> > Thanks
>> >
>> > On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua <si...@datatorrent.com>
>> > wrote:
>> >
>> > > Yes yes,
>> > > We need to put the clear plan upfront
>> > >
>> > > This ticket was thought and scheduled to be done after the
>> > APEXMALHAR-2271
>> > > SpillableSetMultimap which bucketize elements on the time from values.
>> > > But per discussion with David, we realized this ticket is more
>> important
>> > > and affect the general purging functionality so I just jumped to this
>> > issue
>> > > late last week.
>> > > Now the status is the pull request is there. I feel it can be merged
>> > soon.
>> > > https://github.com/apache/apex-malhar/pull/503
>> > >
>> > > Regards,
>> > > Siyuan
>> > >
>> > > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise <th...@apache.org>
>> wrote:
>> > >
>> > > > David,
>> > > >
>> > > > APEXMALHAR-2301 is still open. Are we on track to complete the
>> JIRAs by
>> > > > Monday or should we move to next release?
>> > > >
>> > > > Siyuan,
>> > > >
>> > > > There is a question on the ticket from a month ago that wasn't
>> > > addressed. I
>> > > > think there needs to be more visibility on the thought process and
>> > > > decisions made, otherwise it is very difficult for others to follow.
>> > Open
>> > > > collaboration is key.
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan <da...@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > > Hi Thomas,
>> > > > >
>> > > > > We would like to finish the following tickets if possible for the
>> > 3.6.0
>> > > > > release:
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA:
>> 11/18)
>> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA:
>> 11/21)
>> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA:
>> 11/21)
>> > > > >
>> > > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271, since
>> > this
>> > > > only
>> > > > > affects Session Windows and still quite some work needs to be done
>> > > there,
>> > > > > we would like to defer that to the next release.
>> > > > >
>> > > > > Siyuan and Bright, please also chime in if I'm missing anything.
>> > > > >
>> > > > > David
>> > > > >
>> > > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise <th...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > David,
>> > > > > >
>> > > > > > Any update WRT APEXMALHAR-2130
>> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2130> /
>> > > > > APEXMALHAR-2301
>> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2301> ?
>> > > > > >
>> > > > > > I would like to cut first RC by end of the week.
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise <th...@apache.org>
>> > > wrote:
>> > > > > >
>> > > > > > > It has been a while and the issues that we were waiting for
>> are
>> > > still
>> > > > > > > unresolved.
>> > > > > > >
>> > > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
>> > > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
>> > > > > > 20resolution%20%3D%
>> > > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
>> > > > > > >
>> > > > > > > I would suggest:
>> > > > > > >
>> > > > > > > APEXMALHAR-2300 <https://issues.apache.org/
>> > > > jira/browse/APEXMALHAR-2300
>> > > > > >
>> > > > > > -
>> > > > > > > defer
>> > > > > > > APEXMALHAR-2130 <https://issues.apache.org/
>> > > > jira/browse/APEXMALHAR-2130
>> > > > > >
>> > > > > > /
>> > > > > > > APEXMALHAR-2301 <https://issues.apache.org/
>> > > > jira/browse/APEXMALHAR-2301
>> > > > > >
>> > > > > > -
>> > > > > > > David/Siyuan can you please give an update and recommendation.
>> > > > > > > APEXMALHAR-2284 <https://issues.apache.org/
>> > > > jira/browse/APEXMALHAR-2284
>> > > > > >
>> > > > > > -
>> > > > > > > see discussion on the PR and JIRA. Also discussed it with
>> Bhupesh
>> > > and
>> > > > > > > Chinmay today and we think that this join operator should
>> > probably
>> > > be
>> > > > > > > replaced with the new operator that is based on
>> WindowOperator.
>> > > > > > > APEXMALHAR-2203 <https://issues.apache.org/
>> > > > jira/browse/APEXMALHAR-2203
>> > > > > >
>> > > > > > -
>> > > > > > > defer
>> > > > > > >
>> > > > > > > Please provide any feedback you may have within a day so that
>> we
>> > > can
>> > > > > > > unblock the release.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Thomas
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya Chebolu <
>> > > > > > > chaitanya@datatorrent.com> wrote:
>> > > > > > >
>> > > > > > >> Hi Thomas,
>> > > > > > >>
>> > > > > > >> I am working on APEXMALHAR-2284 and will open a PR in
>> couple
>> > of
>> > > > > days.
>> > > > > > >>
>> > > > > > >> Regards,
>> > > > > > >> Chaitanya
>> > > > > > >>
>> > > > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas Weise <
>> thw@apache.org>
>> > > > wrote:
>> > > > > > >>
>> > > > > > >> > Is anyone working on APEXMALHAR-2284 ?
>> > > > > > >> >
>> > > > > > >> > On Fri, Oct 28, 2016 at 11:00 AM, Bhupesh Chawda <
>> > > > > > >> bhupesh@datatorrent.com>
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> > > +1
>> > > > > > >> > >
>> > > > > > >> > > ~ Bhupesh
>> > > > > > >> > >
>> > > > > > >> > > On Fri, Oct 28, 2016 at 2:29 PM, Tushar Gosavi <
>> > > > > > >> tushar@datatorrent.com>
>> > > > > > >> > > wrote:
>> > > > > > >> > >
>> > > > > > >> > > > +1
>> > > > > > >> > > >
>> > > > > > >> > > > - Tushar.
>> > > > > > >> > > >
>> > > > > > >> > > >
>> > > > > > >> > > > On Fri, Oct 28, 2016 at 12:52 PM, Aniruddha Thombare
>> > > > > > >> > > > <an...@datatorrent.com> wrote:
>> > > > > > >> > > > > +1
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks,
>> > > > > > >> > > > >
>> > > > > > >> > > > > A
>> > > > > > >> > > > >
>> > > > > > >> > > > > _____________________________________
>> > > > > > >> > > > > Sent with difficulty, I mean handheld ;)
>> > > > > > >> > > > >
>> > > > > > >> > > > > On 28 Oct 2016 12:30 pm, "Siyuan Hua" <
>> > > > siyuan@datatorrent.com
>> > > > > >
>> > > > > > >> > wrote:
>> > > > > > >> > > > >
>> > > > > > >> > > > >> +1
>> > > > > > >> > > > >>
>> > > > > > >> > > > >> Thanks！
>> > > > > > >> > > > >>
>> > > > > > >> > > > >> Sent from my iPhone
>> > > > > > >> > > > >>
>> > > > > > >> > > > >> > On Oct 26, 2016, at 13:11, Thomas Weise <
>> > > thw@apache.org>
>> > > > > > >> wrote:
>> > > > > > >> > > > >> >
>> > > > > > >> > > > >> > Hi,
>> > > > > > >> > > > >> >
>> > > > > > >> > > > >> > I'm proposing another release of Malhar in
>> November.
>> > > > There
>> > > > > > are
>> > > > > > >> 49
>> > > > > > >> > > > issues
>> > > > > > >> > > > >> > marked for the release, including important bug
>> > fixes,
>> > > > new
>> > > > > > >> > > > documentation,
>> > > > > > >> > > > >> > SQL support and the work for windowed operator
>> state
>> > > > > > >> management:
>> > > > > > >> > > > >> >
>> > > > > > >> > > > >> > https://issues.apache.org/
>> > jira/issues/?jql=fixVersion%
>> > > > > > >> > > > >> 20%3D%203.6.0%20AND%20project%
>> > > 20%3D%20APEXMALHAR%20ORDER%
>> > > > > > >> > > > >> 20BY%20status%20ASC
>> > > > > > >> > > > >> >
>> > > > > > >> > > > >> > Currently there is at least one blocker, the join
>> > > > operator
>> > > > > is
>> > > > > > >> > broken
>> > > > > > >> > > > >> after
>> > > > > > >> > > > >> > change in managed state. It also affects the SQL
>> > > feature.
>> > > > > > >> > > > >> >
>> > > > > > >> > > > >> > Thanks,
>> > > > > > >> > > > >> > Thomas
>> > > > > > >> > > > >>
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Malhar release 3.6
Posted by Thomas Weise <th...@apache.org>.
Thanks David.
Anyone else wants to contribute something to the release notes? It is time
to wrap it up.
Thanks,
Thomas
On Tue, Dec 6, 2016 at 3:14 PM, David Yan <da...@datatorrent.com> wrote:
> For the work in the Windowed Operator that is included in the 3.6 release:
>
> There is now an option to use spillable data structures for the state
> storage of the Windowed Operator. This enables the Windowed Operator to
> store large states and perform efficient checkpointing.
>
> We also did benchmarking on WindowedOperator with the spillable data
> structures. From the result of our findings, we improved greatly how
> objects are serialized and reduced garbage collection considerably in the
> Managed State layer.
>
> Work is still in progress for purging state that is not needed any more and
> further improving the performance of Managed State that the spillable data
> structures depend on.
>
> Siyuan and Bright, please add to this if I'm missing anything.
>
> David
>
> On Fri, Dec 2, 2016 at 8:55 PM, Thomas Weise <th...@apache.org> wrote:
>
> > The release is available already. For the announcement, would anyone want
> > to provide some information that can go on the blog page, similar to what
> > we had last time:
> >
> > https://blogs.apache.org/apex/entry/apache_apex_malhar_3_5
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Nov 22, 2016 at 2:14 PM, Thomas Weise <th...@apache.org> wrote:
> >
> > > IMO we should not delay the release any longer. I will cut the RC
> > tomorrow
> > > unless there is an objection.
> > >
> > > Thomas
> > >
> > >
> > > On Tue, Nov 22, 2016 at 2:05 PM, David Yan <da...@datatorrent.com>
> > wrote:
> > >
> > >> Thomas,
> > >>
> > >> We need more time for the timebucket change and the purging feature,
> and
> > >> we
> > >> need to do benchmarking after those features have been implemented.
> > >> Is it possible to postpone the release? If not, we can shoot for
> > >> completing
> > >> the features for the 3.7.0 release.
> > >>
> > >> David
> > >>
> > >> On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise <th...@apache.org>
> wrote:
> > >>
> > >> > David, Siyuan,
> > >> >
> > >> > Any update, I would like to cut the RC.
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua <
> siyuan@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > Yes yes,
> > >> > > We need to put the clear plan upfront
> > >> > >
> > >> > > This ticket was thought and scheduled to be done after the
> > >> > APEXMALHAR-2271
> > >> > > SpillableSetMultimap which bucketize elements on the time from
> > values.
> > >> > > But per discussion with David, we realized this ticket is more
> > >> important
> > >> > > and affect the general purging functionality so I just jumped to
> > this
> > >> > issue
> > >> > > late last week.
> > >> > > Now the status is the pull request is there. I feel it can be
> merged
> > >> > soon.
> > >> > > https://github.com/apache/apex-malhar/pull/503
> > >> > >
> > >> > > Regards,
> > >> > > Siyuan
> > >> > >
> > >> > > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise <th...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > David,
> > >> > > >
> > >> > > > APEXMALHAR-2301 is still open. Are we on track to complete the
> > >> JIRAs by
> > >> > > > Monday or should we move to next release?
> > >> > > >
> > >> > > > Siyuan,
> > >> > > >
> > >> > > > There is a question on the ticket from a month ago that wasn't
> > >> > > addressed. I
> > >> > > > think there needs to be more visibility on the thought process
> and
> > >> > > > decisions made, otherwise it is very difficult for others to
> > follow.
> > >> > Open
> > >> > > > collaboration is key.
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan <
> david@datatorrent.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Thomas,
> > >> > > > >
> > >> > > > > We would like to finish the following tickets if possible for
> > the
> > >> > 3.6.0
> > >> > > > > release:
> > >> > > > >
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA:
> > >> 11/18)
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA:
> > >> 11/21)
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA:
> > >> 11/21)
> > >> > > > >
> > >> > > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271,
> > since
> > >> > this
> > >> > > > only
> > >> > > > > affects Session Windows and still quite some work needs to be
> > done
> > >> > > there,
> > >> > > > > we would like to defer that to the next release.
> > >> > > > >
> > >> > > > > Siyuan and Bright, please also chime in if I'm missing
> anything.
> > >> > > > >
> > >> > > > > David
> > >> > > > >
> > >> > > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise <thw@apache.org
> >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > David,
> > >> > > > > >
> > >> > > > > > Any update WRT APEXMALHAR-2130
> > >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2130> /
> > >> > > > > APEXMALHAR-2301
> > >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2301> ?
> > >> > > > > >
> > >> > > > > > I would like to cut first RC by end of the week.
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise <
> > thw@apache.org>
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > > It has been a while and the issues that we were waiting
> for
> > >> are
> > >> > > still
> > >> > > > > > > unresolved.
> > >> > > > > > >
> > >> > > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > >> > > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> > >> > > > > > 20resolution%20%3D%
> > >> > > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> > >> > > > > > >
> > >> > > > > > > I would suggest:
> > >> > > > > > >
> > >> > > > > > > APEXMALHAR-2300 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2300
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > defer
> > >> > > > > > > APEXMALHAR-2130 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2130
> > >> > > > > >
> > >> > > > > > /
> > >> > > > > > > APEXMALHAR-2301 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2301
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > David/Siyuan can you please give an update and
> > recommendation.
> > >> > > > > > > APEXMALHAR-2284 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2284
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > see discussion on the PR and JIRA. Also discussed it with
> > >> Bhupesh
> > >> > > and
> > >> > > > > > > Chinmay today and we think that this join operator should
> > >> > probably
> > >> > > be
> > >> > > > > > > replaced with the new operator that is based on
> > >> WindowOperator.
> > >> > > > > > > APEXMALHAR-2203 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2203
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > defer
> > >> > > > > > >
> > >> > > > > > > Please provide any feedback you may have within a day so
> > that
> > >> we
> > >> > > can
> > >> > > > > > > unblock the release.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Thomas
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya Chebolu <
> > >> > > > > > > chaitanya@datatorrent.com> wrote:
> > >> > > > > > >
> > >> > > > > > >> Hi Thomas,
> > >> > > > > > >>
> > >> > > > > > >> I am working on APEXMALHAR-2284 and will open a PR in
> > >> couple
> > >> > of
> > >> > > > > days.
> > >> > > > > > >>
> > >> > > > > > >> Regards,
> > >> > > > > > >> Chaitanya
> > >> > > > > > >>
> > >> > > > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas Weise <
> > >> thw@apache.org>
> > >> > > > wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Is anyone working on APEXMALHAR-2284 ?
> > >> > > > > > >> >
> > >> > > > > > >> > On Fri, Oct 28, 2016 at 11:00 AM, Bhupesh Chawda <
> > >> > > > > > >> bhupesh@datatorrent.com>
> > >> > > > > > >> > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > +1
> > >> > > > > > >> > >
> > >> > > > > > >> > > ~ Bhupesh
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Fri, Oct 28, 2016 at 2:29 PM, Tushar Gosavi <
> > >> > > > > > >> tushar@datatorrent.com>
> > >> > > > > > >> > > wrote:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > +1
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > - Tushar.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Fri, Oct 28, 2016 at 12:52 PM, Aniruddha
> Thombare
> > >> > > > > > >> > > > <an...@datatorrent.com> wrote:
> > >> > > > > > >> > > > > +1
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > A
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > _____________________________________
> > >> > > > > > >> > > > > Sent with difficulty, I mean handheld ;)
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > On 28 Oct 2016 12:30 pm, "Siyuan Hua" <
> > >> > > > siyuan@datatorrent.com
> > >> > > > > >
> > >> > > > > > >> > wrote:
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > >> +1
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> Thanks！
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> Sent from my iPhone
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> > On Oct 26, 2016, at 13:11, Thomas Weise <
> > >> > > thw@apache.org>
> > >> > > > > > >> wrote:
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Hi,
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > I'm proposing another release of Malhar in
> > >> November.
> > >> > > > There
> > >> > > > > > are
> > >> > > > > > >> 49
> > >> > > > > > >> > > > issues
> > >> > > > > > >> > > > >> > marked for the release, including important
> bug
> > >> > fixes,
> > >> > > > new
> > >> > > > > > >> > > > documentation,
> > >> > > > > > >> > > > >> > SQL support and the work for windowed operator
> > >> state
> > >> > > > > > >> management:
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > https://issues.apache.org/
> > >> > jira/issues/?jql=fixVersion%
> > >> > > > > > >> > > > >> 20%3D%203.6.0%20AND%20project%
> > >> > > 20%3D%20APEXMALHAR%20ORDER%
> > >> > > > > > >> > > > >> 20BY%20status%20ASC
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Currently there is at least one blocker, the
> > join
> > >> > > > operator
> > >> > > > > is
> > >> > > > > > >> > broken
> > >> > > > > > >> > > > >> after
> > >> > > > > > >> > > > >> > change in managed state. It also affects the
> SQL
> > >> > > feature.
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Thanks,
> > >> > > > > > >> > > > >> > Thomas
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Malhar release 3.6
Posted by David Yan <da...@datatorrent.com>.
For the work in the Windowed Operator that is included in the 3.6 release:
There is now an option to use spillable data structures for the state
storage of the Windowed Operator. This enables the Windowed Operator to
store large states and perform efficient checkpointing.
We also did benchmarking on WindowedOperator with the spillable data
structures. From the result of our findings, we improved greatly how
objects are serialized and reduced garbage collection considerably in the
Managed State layer.
Work is still in progress for purging state that is not needed any more and
further improving the performance of Managed State that the spillable data
structures depend on.
Siyuan and Bright, please add to this if I'm missing anything.
David
On Fri, Dec 2, 2016 at 8:55 PM, Thomas Weise <th...@apache.org> wrote:
> The release is available already. For the announcement, would anyone want
> to provide some information that can go on the blog page, similar to what
> we had last time:
>
> https://blogs.apache.org/apex/entry/apache_apex_malhar_3_5
>
> Thanks,
> Thomas
>
>
> On Tue, Nov 22, 2016 at 2:14 PM, Thomas Weise <th...@apache.org> wrote:
>
> > IMO we should not delay the release any longer. I will cut the RC
> tomorrow
> > unless there is an objection.
> >
> > Thomas
> >
> >
> > On Tue, Nov 22, 2016 at 2:05 PM, David Yan <da...@datatorrent.com>
> wrote:
> >
> >> Thomas,
> >>
> >> We need more time for the timebucket change and the purging feature, and
> >> we
> >> need to do benchmarking after those features have been implemented.
> >> Is it possible to postpone the release? If not, we can shoot for
> >> completing
> >> the features for the 3.7.0 release.
> >>
> >> David
> >>
> >> On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise <th...@apache.org> wrote:
> >>
> >> > David, Siyuan,
> >> >
> >> > Any update, I would like to cut the RC.
> >> >
> >> > Thanks
> >> >
> >> > On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua <si...@datatorrent.com>
> >> > wrote:
> >> >
> >> > > Yes yes,
> >> > > We need to put the clear plan upfront
> >> > >
> >> > > This ticket was thought and scheduled to be done after the
> >> > APEXMALHAR-2271
> >> > > SpillableSetMultimap which bucketize elements on the time from
> values.
> >> > > But per discussion with David, we realized this ticket is more
> >> important
> >> > > and affect the general purging functionality so I just jumped to
> this
> >> > issue
> >> > > late last week.
> >> > > Now the status is the pull request is there. I feel it can be merged
> >> > soon.
> >> > > https://github.com/apache/apex-malhar/pull/503
> >> > >
> >> > > Regards,
> >> > > Siyuan
> >> > >
> >> > > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise <th...@apache.org>
> >> wrote:
> >> > >
> >> > > > David,
> >> > > >
> >> > > > APEXMALHAR-2301 is still open. Are we on track to complete the
> >> JIRAs by
> >> > > > Monday or should we move to next release?
> >> > > >
> >> > > > Siyuan,
> >> > > >
> >> > > > There is a question on the ticket from a month ago that wasn't
> >> > > addressed. I
> >> > > > think there needs to be more visibility on the thought process and
> >> > > > decisions made, otherwise it is very difficult for others to
> follow.
> >> > Open
> >> > > > collaboration is key.
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > >
> >> > > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan <david@datatorrent.com
> >
> >> > > wrote:
> >> > > >
> >> > > > > Hi Thomas,
> >> > > > >
> >> > > > > We would like to finish the following tickets if possible for
> the
> >> > 3.6.0
> >> > > > > release:
> >> > > > >
> >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA:
> >> 11/18)
> >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA:
> >> 11/21)
> >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA:
> >> 11/21)
> >> > > > >
> >> > > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271,
> since
> >> > this
> >> > > > only
> >> > > > > affects Session Windows and still quite some work needs to be
> done
> >> > > there,
> >> > > > > we would like to defer that to the next release.
> >> > > > >
> >> > > > > Siyuan and Bright, please also chime in if I'm missing anything.
> >> > > > >
> >> > > > > David
> >> > > > >
> >> > > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise <th...@apache.org>
> >> > wrote:
> >> > > > >
> >> > > > > > David,
> >> > > > > >
> >> > > > > > Any update WRT APEXMALHAR-2130
> >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2130> /
> >> > > > > APEXMALHAR-2301
> >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2301> ?
> >> > > > > >
> >> > > > > > I would like to cut first RC by end of the week.
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > >
> >> > > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise <
> thw@apache.org>
> >> > > wrote:
> >> > > > > >
> >> > > > > > > It has been a while and the issues that we were waiting for
> >> are
> >> > > still
> >> > > > > > > unresolved.
> >> > > > > > >
> >> > > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> >> > > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> >> > > > > > 20resolution%20%3D%
> >> > > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> >> > > > > > >
> >> > > > > > > I would suggest:
> >> > > > > > >
> >> > > > > > > APEXMALHAR-2300 <https://issues.apache.org/
> >> > > > jira/browse/APEXMALHAR-2300
> >> > > > > >
> >> > > > > > -
> >> > > > > > > defer
> >> > > > > > > APEXMALHAR-2130 <https://issues.apache.org/
> >> > > > jira/browse/APEXMALHAR-2130
> >> > > > > >
> >> > > > > > /
> >> > > > > > > APEXMALHAR-2301 <https://issues.apache.org/
> >> > > > jira/browse/APEXMALHAR-2301
> >> > > > > >
> >> > > > > > -
> >> > > > > > > David/Siyuan can you please give an update and
> recommendation.
> >> > > > > > > APEXMALHAR-2284 <https://issues.apache.org/
> >> > > > jira/browse/APEXMALHAR-2284
> >> > > > > >
> >> > > > > > -
> >> > > > > > > see discussion on the PR and JIRA. Also discussed it with
> >> Bhupesh
> >> > > and
> >> > > > > > > Chinmay today and we think that this join operator should
> >> > probably
> >> > > be
> >> > > > > > > replaced with the new operator that is based on
> >> WindowOperator.
> >> > > > > > > APEXMALHAR-2203 <https://issues.apache.org/
> >> > > > jira/browse/APEXMALHAR-2203
> >> > > > > >
> >> > > > > > -
> >> > > > > > > defer
> >> > > > > > >
> >> > > > > > > Please provide any feedback you may have within a day so
> that
> >> we
> >> > > can
> >> > > > > > > unblock the release.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Thomas
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya Chebolu <
> >> > > > > > > chaitanya@datatorrent.com> wrote:
> >> > > > > > >
> >> > > > > > >> Hi Thomas,
> >> > > > > > >>
> >> > > > > > >> I am working on APEXMALHAR-2284 and will open a PR in
> >> couple
> >> > of
> >> > > > > days.
> >> > > > > > >>
> >> > > > > > >> Regards,
> >> > > > > > >> Chaitanya
> >> > > > > > >>
> >> > > > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas Weise <
> >> thw@apache.org>
> >> > > > wrote:
> >> > > > > > >>
> >> > > > > > >> > Is anyone working on APEXMALHAR-2284 ?
> >> > > > > > >> >
> >> > > > > > >> > On Fri, Oct 28, 2016 at 11:00 AM, Bhupesh Chawda <
> >> > > > > > >> bhupesh@datatorrent.com>
> >> > > > > > >> > wrote:
> >> > > > > > >> >
> >> > > > > > >> > > +1
> >> > > > > > >> > >
> >> > > > > > >> > > ~ Bhupesh
> >> > > > > > >> > >
> >> > > > > > >> > > On Fri, Oct 28, 2016 at 2:29 PM, Tushar Gosavi <
> >> > > > > > >> tushar@datatorrent.com>
> >> > > > > > >> > > wrote:
> >> > > > > > >> > >
> >> > > > > > >> > > > +1
> >> > > > > > >> > > >
> >> > > > > > >> > > > - Tushar.
> >> > > > > > >> > > >
> >> > > > > > >> > > >
> >> > > > > > >> > > > On Fri, Oct 28, 2016 at 12:52 PM, Aniruddha Thombare
> >> > > > > > >> > > > <an...@datatorrent.com> wrote:
> >> > > > > > >> > > > > +1
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Thanks,
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > A
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > _____________________________________
> >> > > > > > >> > > > > Sent with difficulty, I mean handheld ;)
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > On 28 Oct 2016 12:30 pm, "Siyuan Hua" <
> >> > > > siyuan@datatorrent.com
> >> > > > > >
> >> > > > > > >> > wrote:
> >> > > > > > >> > > > >
> >> > > > > > >> > > > >> +1
> >> > > > > > >> > > > >>
> >> > > > > > >> > > > >> Thanks！
> >> > > > > > >> > > > >>
> >> > > > > > >> > > > >> Sent from my iPhone
> >> > > > > > >> > > > >>
> >> > > > > > >> > > > >> > On Oct 26, 2016, at 13:11, Thomas Weise <
> >> > > thw@apache.org>
> >> > > > > > >> wrote:
> >> > > > > > >> > > > >> >
> >> > > > > > >> > > > >> > Hi,
> >> > > > > > >> > > > >> >
> >> > > > > > >> > > > >> > I'm proposing another release of Malhar in
> >> November.
> >> > > > There
> >> > > > > > are
> >> > > > > > >> 49
> >> > > > > > >> > > > issues
> >> > > > > > >> > > > >> > marked for the release, including important bug
> >> > fixes,
> >> > > > new
> >> > > > > > >> > > > documentation,
> >> > > > > > >> > > > >> > SQL support and the work for windowed operator
> >> state
> >> > > > > > >> management:
> >> > > > > > >> > > > >> >
> >> > > > > > >> > > > >> > https://issues.apache.org/
> >> > jira/issues/?jql=fixVersion%
> >> > > > > > >> > > > >> 20%3D%203.6.0%20AND%20project%
> >> > > 20%3D%20APEXMALHAR%20ORDER%
> >> > > > > > >> > > > >> 20BY%20status%20ASC
> >> > > > > > >> > > > >> >
> >> > > > > > >> > > > >> > Currently there is at least one blocker, the
> join
> >> > > > operator
> >> > > > > is
> >> > > > > > >> > broken
> >> > > > > > >> > > > >> after
> >> > > > > > >> > > > >> > change in managed state. It also affects the SQL
> >> > > feature.
> >> > > > > > >> > > > >> >
> >> > > > > > >> > > > >> > Thanks,
> >> > > > > > >> > > > >> > Thomas
> >> > > > > > >> > > > >>
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>