You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Xinyu Liu <xi...@gmail.com> on 2018/04/30 19:10:02 UTC
[Discuss] Samza 0.14.1 release
Hi, All,
We have been adding many improvements and critical bug fixes in the areas
of Samza sql, standalone, eventhub system consumer and host-affinity since
0.14.0 release. The changes should warrant a new minor release:
SQL
- SAMZA-1681: Samza-sql: Add support for handling older record schema
versions in AvroRelConverter
- SAMZA-1671: SamzaSQL: add insert into table support
- SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
Standalone
- SAMZA-1689: Add validations before state transitions in
ZkBarrierForVersionUpgrade.
- SAMZA-1686: Set finite operation timeout when creating zkClient.
- SAMZA-1667: Skip storing configuration as a part of JobModel in zookeeper
data nodes.
- SAMZA-1647: Fix NPE in JobModelExpired event handler.
Eventhub
- SAMZA-1688: use per partition eventhubs client
- SAMZA-1676: miscellaneous fix and improvement for eventhubs system
- SAMZA-1656: EventHubSystemAdmin does not fetch metadata for valid streams.
host-affinity
- SAMZA-1687: Prioritize preferred host requests over ANY-HOST requests
- SAMZA-1649: Improve host-aware allocation to account for strict locality
The complete list of changes are here:
https://issues.apache.org/jira/browse/SAMZA-1624?jql=project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.
Most JIRAs in the list have been completed and merged, and we willl try to
get the remaining completed before 0.14.1 release.
Here's what I propose:
- Cut the 0.14.1 release branch
- Target a release sometime next week (5/7 - 5/11).
Thoughts?
Xinyu
Re: [Discuss] Samza 0.14.1 release
Posted by Boris S <bo...@gmail.com>.
Sure. Let's do it.
On Wed, May 2, 2018 at 9:58 AM, Xinyu Liu <xi...@gmail.com> wrote:
> Thanks everyone for the discussion! I marked the Kafka 0.11 upgrade ticket
> as 0.14.1 and will include it in this release. Now we have 3 binding votes
> from Jagadish, Yi and Jake, so I will go ahead and start the release
> procedure.
>
> Thanks,
> Xinyu
>
> On Tue, May 1, 2018 at 4:31 PM, Jake Maes <jm...@apache.org> wrote:
>
> > Yes, lets do it.
> >
> > On Tue, May 1, 2018 at 11:55 AM, Tom Davis <to...@recursivedream.com>
> wrote:
> >
> > > Yeah, I'd love that!
> > >
> > >
> > > On Tue, May 1, 2018, at 1:05 PM, Yi Pan wrote:
> > > > Yes, for Kafka 0.11 change, it mainly just affects the Kafka system
> > > > factory, admin, consumer, and producer's implementation changes.
> > > > There is> no API changes that affects the user code. The current
> > version
> > > > on master> should be source-compatible w/ existing Samza applications
> > on
> > > > Samza 0.14.0> and only needs re-compile and re-package with the new
> > > Kafka system
> > > > implementations.
> > > >
> > > > @Tom, does that sounds good to you if we include that as 0.14.1?
> > > >
> > > > -Yi
> > > >
> > > > On Mon, Apr 30, 2018 at 6:09 PM, Xinyu Liu
> > > > <xi...@gmail.com> wrote:>
> > > >> I chatted with Yi offline and seems for 0.11.0 Kafka upgrade,
> > > >> there is no>> user code change required. The changes we are aware of
> > > are that
> > > >> some of the>> old producer configs deprecated in 0.10 are finally
> > > removed in this
> > > >> version>> (e.g. block.on.buffer.full). If there are no objections, I
> > am
> > > >> going to>> include it in this release too.
> > > >>
> > > >> Thanks,
> > > >> Xinyu
> > > >>
> > > >> On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
> > > >> jagadish1989@gmail.com> wrote:
> > > >>
> > > >>> Xinyu,
> > > >>> This comes up with an exciting lineup of features. +1 (binding)
> > > >>> from my>>> side.
> > > >>>
> > > >>> Tom,
> > > >>>
> > > >>>>> These all look like excellent changes to me, I'd be happy to have
> > > >>>>> more>>> stability in Standalone!
> > > >>>
> > > >>> We are actively investing in standalone and have fixed some
> > > >>> edge-cases>> that
> > > >>> surfaced when running it at scale. Thanks much for the feedback!
> > > >>>
> > > >>>>> Any timeline on getting Kafka 0.11 upgrade in?
> > > >>>
> > > >>> We have historically been doing Kafka upgrades on major releases
> > > >>> (instead>>> of
> > > >>> minor ones). So, it will likely be in 0.15.0. I'll provide an
> > > >>> update on>> the
> > > >>> timeline
> > > >>> soon.
> > > >>>
> > > >>> Best,
> > > >>> Jagadish
> > > >>>
> > > >>>
> > > >>> On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <tom@recursivedream.com
> > >>>
> > > wrote:
> > > >>>
> > > >>>> These all look like excellent changes to me, I'd be happy to
> > > >>>> have more>>>> stability in Standalone!
> > > >>>> Any timeline on getting Kafka 0.11 upgrade in? It seems to
> > > >>>> have been>>>> done a while, but waiting for 0.15 I presume.
> > > >>>> Best,
> > > >>>>
> > > >>>> Tom
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> > > >>>>> Hi, All,
> > > >>>>>
> > > >>>>> We have been adding many improvements and critical bug fixes in
> > > >>>>> the areas> of Samza sql, standalone, eventhub system consumer
> and>>
> > > host-
> > > >>>>> affinity since> 0.14.0 release. The changes should warrant a new
> > > >> minor
> > > >>>> release:
> > > >>>>>
> > > >>>>> SQL
> > > >>>>> - SAMZA-1681: Samza-sql: Add support for handling older record
> > > >> schema>
> > > >>>> versions in AvroRelConverter
> > > >>>>> - SAMZA-1671: SamzaSQL: add insert into table support
> > > >>>>> - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> > > >>>>>
> > > >>>>> Standalone
> > > >>>>> - SAMZA-1689: Add validations before state transitions in
> > > >>>>> ZkBarrierForVersionUpgrade.
> > > >>>>> - SAMZA-1686: Set finite operation timeout when creating
> > > zkClient.>>>>> - SAMZA-1667: Skip storing configuration as a part of
> > > JobModel in>>>>> zookeeper> data nodes.
> > > >>>>> - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> > > >>>>>
> > > >>>>> Eventhub
> > > >>>>> - SAMZA-1688: use per partition eventhubs client
> > > >>>>> - SAMZA-1676: miscellaneous fix and improvement for eventhubs
> > > >>>>> system>>>>> - SAMZA-1656: EventHubSystemAdmin does not fetch
> > > metadata for
> > > >>>>> valid>>>>> streams.>
> > > >>>>> host-affinity
> > > >>>>> - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> > > >>>>> requests> - SAMZA-1649: Improve host-aware allocation to account
> > > >> for
> > > >>>> strict
> > > >>>>> locality>
> > > >>>>> The complete list of changes are here:
> > > >>>>> https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> > > >>>> project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> > > >>>> 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most
> JIRAs
> > > >>>> in>>> the
> > > >>>> list have been completed and merged, and we
> > > >>>>> willl try to> get the remaining completed before 0.14.1 release.
> > > >>>>>
> > > >>>>> Here's what I propose:
> > > >>>>> - Cut the 0.14.1 release branch
> > > >>>>> - Target a release sometime next week (5/7 - 5/11).
> > > >>>>> Thoughts?
> > > >>>>>
> > > >>>>> Xinyu
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Jagadish V,
> > > >>> Graduate Student,
> > > >>> Department of Computer Science,
> > > >>> Stanford University
> > > >>>
> > > >>
> > >
> >
>
Re: [Discuss] Samza 0.14.1 release
Posted by Xinyu Liu <xi...@gmail.com>.
Thanks everyone for the discussion! I marked the Kafka 0.11 upgrade ticket
as 0.14.1 and will include it in this release. Now we have 3 binding votes
from Jagadish, Yi and Jake, so I will go ahead and start the release
procedure.
Thanks,
Xinyu
On Tue, May 1, 2018 at 4:31 PM, Jake Maes <jm...@apache.org> wrote:
> Yes, lets do it.
>
> On Tue, May 1, 2018 at 11:55 AM, Tom Davis <to...@recursivedream.com> wrote:
>
> > Yeah, I'd love that!
> >
> >
> > On Tue, May 1, 2018, at 1:05 PM, Yi Pan wrote:
> > > Yes, for Kafka 0.11 change, it mainly just affects the Kafka system
> > > factory, admin, consumer, and producer's implementation changes.
> > > There is> no API changes that affects the user code. The current
> version
> > > on master> should be source-compatible w/ existing Samza applications
> on
> > > Samza 0.14.0> and only needs re-compile and re-package with the new
> > Kafka system
> > > implementations.
> > >
> > > @Tom, does that sounds good to you if we include that as 0.14.1?
> > >
> > > -Yi
> > >
> > > On Mon, Apr 30, 2018 at 6:09 PM, Xinyu Liu
> > > <xi...@gmail.com> wrote:>
> > >> I chatted with Yi offline and seems for 0.11.0 Kafka upgrade,
> > >> there is no>> user code change required. The changes we are aware of
> > are that
> > >> some of the>> old producer configs deprecated in 0.10 are finally
> > removed in this
> > >> version>> (e.g. block.on.buffer.full). If there are no objections, I
> am
> > >> going to>> include it in this release too.
> > >>
> > >> Thanks,
> > >> Xinyu
> > >>
> > >> On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
> > >> jagadish1989@gmail.com> wrote:
> > >>
> > >>> Xinyu,
> > >>> This comes up with an exciting lineup of features. +1 (binding)
> > >>> from my>>> side.
> > >>>
> > >>> Tom,
> > >>>
> > >>>>> These all look like excellent changes to me, I'd be happy to have
> > >>>>> more>>> stability in Standalone!
> > >>>
> > >>> We are actively investing in standalone and have fixed some
> > >>> edge-cases>> that
> > >>> surfaced when running it at scale. Thanks much for the feedback!
> > >>>
> > >>>>> Any timeline on getting Kafka 0.11 upgrade in?
> > >>>
> > >>> We have historically been doing Kafka upgrades on major releases
> > >>> (instead>>> of
> > >>> minor ones). So, it will likely be in 0.15.0. I'll provide an
> > >>> update on>> the
> > >>> timeline
> > >>> soon.
> > >>>
> > >>> Best,
> > >>> Jagadish
> > >>>
> > >>>
> > >>> On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <tom@recursivedream.com
> >>>
> > wrote:
> > >>>
> > >>>> These all look like excellent changes to me, I'd be happy to
> > >>>> have more>>>> stability in Standalone!
> > >>>> Any timeline on getting Kafka 0.11 upgrade in? It seems to
> > >>>> have been>>>> done a while, but waiting for 0.15 I presume.
> > >>>> Best,
> > >>>>
> > >>>> Tom
> > >>>>
> > >>>>
> > >>>> On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> > >>>>> Hi, All,
> > >>>>>
> > >>>>> We have been adding many improvements and critical bug fixes in
> > >>>>> the areas> of Samza sql, standalone, eventhub system consumer and>>
> > host-
> > >>>>> affinity since> 0.14.0 release. The changes should warrant a new
> > >> minor
> > >>>> release:
> > >>>>>
> > >>>>> SQL
> > >>>>> - SAMZA-1681: Samza-sql: Add support for handling older record
> > >> schema>
> > >>>> versions in AvroRelConverter
> > >>>>> - SAMZA-1671: SamzaSQL: add insert into table support
> > >>>>> - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> > >>>>>
> > >>>>> Standalone
> > >>>>> - SAMZA-1689: Add validations before state transitions in
> > >>>>> ZkBarrierForVersionUpgrade.
> > >>>>> - SAMZA-1686: Set finite operation timeout when creating
> > zkClient.>>>>> - SAMZA-1667: Skip storing configuration as a part of
> > JobModel in>>>>> zookeeper> data nodes.
> > >>>>> - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> > >>>>>
> > >>>>> Eventhub
> > >>>>> - SAMZA-1688: use per partition eventhubs client
> > >>>>> - SAMZA-1676: miscellaneous fix and improvement for eventhubs
> > >>>>> system>>>>> - SAMZA-1656: EventHubSystemAdmin does not fetch
> > metadata for
> > >>>>> valid>>>>> streams.>
> > >>>>> host-affinity
> > >>>>> - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> > >>>>> requests> - SAMZA-1649: Improve host-aware allocation to account
> > >> for
> > >>>> strict
> > >>>>> locality>
> > >>>>> The complete list of changes are here:
> > >>>>> https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> > >>>> project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> > >>>> 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs
> > >>>> in>>> the
> > >>>> list have been completed and merged, and we
> > >>>>> willl try to> get the remaining completed before 0.14.1 release.
> > >>>>>
> > >>>>> Here's what I propose:
> > >>>>> - Cut the 0.14.1 release branch
> > >>>>> - Target a release sometime next week (5/7 - 5/11).
> > >>>>> Thoughts?
> > >>>>>
> > >>>>> Xinyu
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Jagadish V,
> > >>> Graduate Student,
> > >>> Department of Computer Science,
> > >>> Stanford University
> > >>>
> > >>
> >
>
Re: [Discuss] Samza 0.14.1 release
Posted by Jake Maes <jm...@apache.org>.
Yes, lets do it.
On Tue, May 1, 2018 at 11:55 AM, Tom Davis <to...@recursivedream.com> wrote:
> Yeah, I'd love that!
>
>
> On Tue, May 1, 2018, at 1:05 PM, Yi Pan wrote:
> > Yes, for Kafka 0.11 change, it mainly just affects the Kafka system
> > factory, admin, consumer, and producer's implementation changes.
> > There is> no API changes that affects the user code. The current version
> > on master> should be source-compatible w/ existing Samza applications on
> > Samza 0.14.0> and only needs re-compile and re-package with the new
> Kafka system
> > implementations.
> >
> > @Tom, does that sounds good to you if we include that as 0.14.1?
> >
> > -Yi
> >
> > On Mon, Apr 30, 2018 at 6:09 PM, Xinyu Liu
> > <xi...@gmail.com> wrote:>
> >> I chatted with Yi offline and seems for 0.11.0 Kafka upgrade,
> >> there is no>> user code change required. The changes we are aware of
> are that
> >> some of the>> old producer configs deprecated in 0.10 are finally
> removed in this
> >> version>> (e.g. block.on.buffer.full). If there are no objections, I am
> >> going to>> include it in this release too.
> >>
> >> Thanks,
> >> Xinyu
> >>
> >> On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
> >> jagadish1989@gmail.com> wrote:
> >>
> >>> Xinyu,
> >>> This comes up with an exciting lineup of features. +1 (binding)
> >>> from my>>> side.
> >>>
> >>> Tom,
> >>>
> >>>>> These all look like excellent changes to me, I'd be happy to have
> >>>>> more>>> stability in Standalone!
> >>>
> >>> We are actively investing in standalone and have fixed some
> >>> edge-cases>> that
> >>> surfaced when running it at scale. Thanks much for the feedback!
> >>>
> >>>>> Any timeline on getting Kafka 0.11 upgrade in?
> >>>
> >>> We have historically been doing Kafka upgrades on major releases
> >>> (instead>>> of
> >>> minor ones). So, it will likely be in 0.15.0. I'll provide an
> >>> update on>> the
> >>> timeline
> >>> soon.
> >>>
> >>> Best,
> >>> Jagadish
> >>>
> >>>
> >>> On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <to...@recursivedream.com>>>
> wrote:
> >>>
> >>>> These all look like excellent changes to me, I'd be happy to
> >>>> have more>>>> stability in Standalone!
> >>>> Any timeline on getting Kafka 0.11 upgrade in? It seems to
> >>>> have been>>>> done a while, but waiting for 0.15 I presume.
> >>>> Best,
> >>>>
> >>>> Tom
> >>>>
> >>>>
> >>>> On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> >>>>> Hi, All,
> >>>>>
> >>>>> We have been adding many improvements and critical bug fixes in
> >>>>> the areas> of Samza sql, standalone, eventhub system consumer and>>
> host-
> >>>>> affinity since> 0.14.0 release. The changes should warrant a new
> >> minor
> >>>> release:
> >>>>>
> >>>>> SQL
> >>>>> - SAMZA-1681: Samza-sql: Add support for handling older record
> >> schema>
> >>>> versions in AvroRelConverter
> >>>>> - SAMZA-1671: SamzaSQL: add insert into table support
> >>>>> - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> >>>>>
> >>>>> Standalone
> >>>>> - SAMZA-1689: Add validations before state transitions in
> >>>>> ZkBarrierForVersionUpgrade.
> >>>>> - SAMZA-1686: Set finite operation timeout when creating
> zkClient.>>>>> - SAMZA-1667: Skip storing configuration as a part of
> JobModel in>>>>> zookeeper> data nodes.
> >>>>> - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> >>>>>
> >>>>> Eventhub
> >>>>> - SAMZA-1688: use per partition eventhubs client
> >>>>> - SAMZA-1676: miscellaneous fix and improvement for eventhubs
> >>>>> system>>>>> - SAMZA-1656: EventHubSystemAdmin does not fetch
> metadata for
> >>>>> valid>>>>> streams.>
> >>>>> host-affinity
> >>>>> - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> >>>>> requests> - SAMZA-1649: Improve host-aware allocation to account
> >> for
> >>>> strict
> >>>>> locality>
> >>>>> The complete list of changes are here:
> >>>>> https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> >>>> project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> >>>> 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs
> >>>> in>>> the
> >>>> list have been completed and merged, and we
> >>>>> willl try to> get the remaining completed before 0.14.1 release.
> >>>>>
> >>>>> Here's what I propose:
> >>>>> - Cut the 0.14.1 release branch
> >>>>> - Target a release sometime next week (5/7 - 5/11).
> >>>>> Thoughts?
> >>>>>
> >>>>> Xinyu
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Jagadish V,
> >>> Graduate Student,
> >>> Department of Computer Science,
> >>> Stanford University
> >>>
> >>
>
Re: [Discuss] Samza 0.14.1 release
Posted by Tom Davis <to...@recursivedream.com>.
Yeah, I'd love that!
On Tue, May 1, 2018, at 1:05 PM, Yi Pan wrote:
> Yes, for Kafka 0.11 change, it mainly just affects the Kafka system
> factory, admin, consumer, and producer's implementation changes.
> There is> no API changes that affects the user code. The current version
> on master> should be source-compatible w/ existing Samza applications on
> Samza 0.14.0> and only needs re-compile and re-package with the new Kafka system
> implementations.
>
> @Tom, does that sounds good to you if we include that as 0.14.1?
>
> -Yi
>
> On Mon, Apr 30, 2018 at 6:09 PM, Xinyu Liu
> <xi...@gmail.com> wrote:>
>> I chatted with Yi offline and seems for 0.11.0 Kafka upgrade,
>> there is no>> user code change required. The changes we are aware of are that
>> some of the>> old producer configs deprecated in 0.10 are finally removed in this
>> version>> (e.g. block.on.buffer.full). If there are no objections, I am
>> going to>> include it in this release too.
>>
>> Thanks,
>> Xinyu
>>
>> On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
>> jagadish1989@gmail.com> wrote:
>>
>>> Xinyu,
>>> This comes up with an exciting lineup of features. +1 (binding)
>>> from my>>> side.
>>>
>>> Tom,
>>>
>>>>> These all look like excellent changes to me, I'd be happy to have
>>>>> more>>> stability in Standalone!
>>>
>>> We are actively investing in standalone and have fixed some
>>> edge-cases>> that
>>> surfaced when running it at scale. Thanks much for the feedback!
>>>
>>>>> Any timeline on getting Kafka 0.11 upgrade in?
>>>
>>> We have historically been doing Kafka upgrades on major releases
>>> (instead>>> of
>>> minor ones). So, it will likely be in 0.15.0. I'll provide an
>>> update on>> the
>>> timeline
>>> soon.
>>>
>>> Best,
>>> Jagadish
>>>
>>>
>>> On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <to...@recursivedream.com>>> wrote:
>>>
>>>> These all look like excellent changes to me, I'd be happy to
>>>> have more>>>> stability in Standalone!
>>>> Any timeline on getting Kafka 0.11 upgrade in? It seems to
>>>> have been>>>> done a while, but waiting for 0.15 I presume.
>>>> Best,
>>>>
>>>> Tom
>>>>
>>>>
>>>> On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
>>>>> Hi, All,
>>>>>
>>>>> We have been adding many improvements and critical bug fixes in
>>>>> the areas> of Samza sql, standalone, eventhub system consumer and>> host-
>>>>> affinity since> 0.14.0 release. The changes should warrant a new
>> minor
>>>> release:
>>>>>
>>>>> SQL
>>>>> - SAMZA-1681: Samza-sql: Add support for handling older record
>> schema>
>>>> versions in AvroRelConverter
>>>>> - SAMZA-1671: SamzaSQL: add insert into table support
>>>>> - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
>>>>>
>>>>> Standalone
>>>>> - SAMZA-1689: Add validations before state transitions in
>>>>> ZkBarrierForVersionUpgrade.
>>>>> - SAMZA-1686: Set finite operation timeout when creating zkClient.>>>>> - SAMZA-1667: Skip storing configuration as a part of JobModel in>>>>> zookeeper> data nodes.
>>>>> - SAMZA-1647: Fix NPE in JobModelExpired event handler.
>>>>>
>>>>> Eventhub
>>>>> - SAMZA-1688: use per partition eventhubs client
>>>>> - SAMZA-1676: miscellaneous fix and improvement for eventhubs
>>>>> system>>>>> - SAMZA-1656: EventHubSystemAdmin does not fetch metadata for
>>>>> valid>>>>> streams.>
>>>>> host-affinity
>>>>> - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
>>>>> requests> - SAMZA-1649: Improve host-aware allocation to account
>> for
>>>> strict
>>>>> locality>
>>>>> The complete list of changes are here:
>>>>> https://issues.apache.org/jira/browse/SAMZA-1624?jql=
>>>> project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
>>>> 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs
>>>> in>>> the
>>>> list have been completed and merged, and we
>>>>> willl try to> get the remaining completed before 0.14.1 release.
>>>>>
>>>>> Here's what I propose:
>>>>> - Cut the 0.14.1 release branch
>>>>> - Target a release sometime next week (5/7 - 5/11).
>>>>> Thoughts?
>>>>>
>>>>> Xinyu
>>>>
>>>>
>>>
>>>
>>> --
>>> Jagadish V,
>>> Graduate Student,
>>> Department of Computer Science,
>>> Stanford University
>>>
>>
Re: [Discuss] Samza 0.14.1 release
Posted by Yi Pan <ni...@gmail.com>.
Yes, for Kafka 0.11 change, it mainly just affects the Kafka system
factory, admin, consumer, and producer's implementation changes. There is
no API changes that affects the user code. The current version on master
should be source-compatible w/ existing Samza applications on Samza 0.14.0
and only needs re-compile and re-package with the new Kafka system
implementations.
@Tom, does that sounds good to you if we include that as 0.14.1?
-Yi
On Mon, Apr 30, 2018 at 6:09 PM, Xinyu Liu <xi...@gmail.com> wrote:
> I chatted with Yi offline and seems for 0.11.0 Kafka upgrade, there is no
> user code change required. The changes we are aware of are that some of the
> old producer configs deprecated in 0.10 are finally removed in this version
> (e.g. block.on.buffer.full). If there are no objections, I am going to
> include it in this release too.
>
> Thanks,
> Xinyu
>
> On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
> jagadish1989@gmail.com> wrote:
>
> > Xinyu,
> > This comes up with an exciting lineup of features. +1 (binding) from my
> > side.
> >
> > Tom,
> >
> > >> These all look like excellent changes to me, I'd be happy to have more
> > stability in Standalone!
> >
> > We are actively investing in standalone and have fixed some edge-cases
> that
> > surfaced when running it at scale. Thanks much for the feedback!
> >
> > >> Any timeline on getting Kafka 0.11 upgrade in?
> >
> > We have historically been doing Kafka upgrades on major releases (instead
> > of
> > minor ones). So, it will likely be in 0.15.0. I'll provide an update on
> the
> > timeline
> > soon.
> >
> > Best,
> > Jagadish
> >
> >
> > On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <to...@recursivedream.com>
> wrote:
> >
> > > These all look like excellent changes to me, I'd be happy to have more
> > > stability in Standalone!
> > > Any timeline on getting Kafka 0.11 upgrade in? It seems to have been
> > > done a while, but waiting for 0.15 I presume.
> > > Best,
> > >
> > > Tom
> > >
> > >
> > > On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> > > > Hi, All,
> > > >
> > > > We have been adding many improvements and critical bug fixes in
> > > > the areas> of Samza sql, standalone, eventhub system consumer and
> host-
> > > > affinity since> 0.14.0 release. The changes should warrant a new
> minor
> > > release:
> > > >
> > > > SQL
> > > > - SAMZA-1681: Samza-sql: Add support for handling older record
> schema>
> > > versions in AvroRelConverter
> > > > - SAMZA-1671: SamzaSQL: add insert into table support
> > > > - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> > > >
> > > > Standalone
> > > > - SAMZA-1689: Add validations before state transitions in
> > > > ZkBarrierForVersionUpgrade.
> > > > - SAMZA-1686: Set finite operation timeout when creating zkClient.
> > > > - SAMZA-1667: Skip storing configuration as a part of JobModel in
> > > > zookeeper> data nodes.
> > > > - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> > > >
> > > > Eventhub
> > > > - SAMZA-1688: use per partition eventhubs client
> > > > - SAMZA-1676: miscellaneous fix and improvement for eventhubs system
> > > > - SAMZA-1656: EventHubSystemAdmin does not fetch metadata for valid
> > > > streams.>
> > > > host-affinity
> > > > - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> > > > requests> - SAMZA-1649: Improve host-aware allocation to account
> for
> > > strict
> > > > locality>
> > > > The complete list of changes are here:
> > > > https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> > > project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> > > 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs in
> > the
> > > list have been completed and merged, and we
> > > > willl try to> get the remaining completed before 0.14.1 release.
> > > >
> > > > Here's what I propose:
> > > > - Cut the 0.14.1 release branch
> > > > - Target a release sometime next week (5/7 - 5/11).
> > > > Thoughts?
> > > >
> > > > Xinyu
> > >
> > >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>
Re: [Discuss] Samza 0.14.1 release
Posted by Xinyu Liu <xi...@gmail.com>.
I chatted with Yi offline and seems for 0.11.0 Kafka upgrade, there is no
user code change required. The changes we are aware of are that some of the
old producer configs deprecated in 0.10 are finally removed in this version
(e.g. block.on.buffer.full). If there are no objections, I am going to
include it in this release too.
Thanks,
Xinyu
On Mon, Apr 30, 2018 at 3:49 PM, Jagadish Venkatraman <
jagadish1989@gmail.com> wrote:
> Xinyu,
> This comes up with an exciting lineup of features. +1 (binding) from my
> side.
>
> Tom,
>
> >> These all look like excellent changes to me, I'd be happy to have more
> stability in Standalone!
>
> We are actively investing in standalone and have fixed some edge-cases that
> surfaced when running it at scale. Thanks much for the feedback!
>
> >> Any timeline on getting Kafka 0.11 upgrade in?
>
> We have historically been doing Kafka upgrades on major releases (instead
> of
> minor ones). So, it will likely be in 0.15.0. I'll provide an update on the
> timeline
> soon.
>
> Best,
> Jagadish
>
>
> On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <to...@recursivedream.com> wrote:
>
> > These all look like excellent changes to me, I'd be happy to have more
> > stability in Standalone!
> > Any timeline on getting Kafka 0.11 upgrade in? It seems to have been
> > done a while, but waiting for 0.15 I presume.
> > Best,
> >
> > Tom
> >
> >
> > On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> > > Hi, All,
> > >
> > > We have been adding many improvements and critical bug fixes in
> > > the areas> of Samza sql, standalone, eventhub system consumer and host-
> > > affinity since> 0.14.0 release. The changes should warrant a new minor
> > release:
> > >
> > > SQL
> > > - SAMZA-1681: Samza-sql: Add support for handling older record schema>
> > versions in AvroRelConverter
> > > - SAMZA-1671: SamzaSQL: add insert into table support
> > > - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> > >
> > > Standalone
> > > - SAMZA-1689: Add validations before state transitions in
> > > ZkBarrierForVersionUpgrade.
> > > - SAMZA-1686: Set finite operation timeout when creating zkClient.
> > > - SAMZA-1667: Skip storing configuration as a part of JobModel in
> > > zookeeper> data nodes.
> > > - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> > >
> > > Eventhub
> > > - SAMZA-1688: use per partition eventhubs client
> > > - SAMZA-1676: miscellaneous fix and improvement for eventhubs system
> > > - SAMZA-1656: EventHubSystemAdmin does not fetch metadata for valid
> > > streams.>
> > > host-affinity
> > > - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> > > requests> - SAMZA-1649: Improve host-aware allocation to account for
> > strict
> > > locality>
> > > The complete list of changes are here:
> > > https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> > project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> > 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs in
> the
> > list have been completed and merged, and we
> > > willl try to> get the remaining completed before 0.14.1 release.
> > >
> > > Here's what I propose:
> > > - Cut the 0.14.1 release branch
> > > - Target a release sometime next week (5/7 - 5/11).
> > > Thoughts?
> > >
> > > Xinyu
> >
> >
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>
Re: [Discuss] Samza 0.14.1 release
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Xinyu,
This comes up with an exciting lineup of features. +1 (binding) from my
side.
Tom,
>> These all look like excellent changes to me, I'd be happy to have more
stability in Standalone!
We are actively investing in standalone and have fixed some edge-cases that
surfaced when running it at scale. Thanks much for the feedback!
>> Any timeline on getting Kafka 0.11 upgrade in?
We have historically been doing Kafka upgrades on major releases (instead
of
minor ones). So, it will likely be in 0.15.0. I'll provide an update on the
timeline
soon.
Best,
Jagadish
On Mon, Apr 30, 2018 at 3:18 PM, Tom Davis <to...@recursivedream.com> wrote:
> These all look like excellent changes to me, I'd be happy to have more
> stability in Standalone!
> Any timeline on getting Kafka 0.11 upgrade in? It seems to have been
> done a while, but waiting for 0.15 I presume.
> Best,
>
> Tom
>
>
> On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> > Hi, All,
> >
> > We have been adding many improvements and critical bug fixes in
> > the areas> of Samza sql, standalone, eventhub system consumer and host-
> > affinity since> 0.14.0 release. The changes should warrant a new minor
> release:
> >
> > SQL
> > - SAMZA-1681: Samza-sql: Add support for handling older record schema>
> versions in AvroRelConverter
> > - SAMZA-1671: SamzaSQL: add insert into table support
> > - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
> >
> > Standalone
> > - SAMZA-1689: Add validations before state transitions in
> > ZkBarrierForVersionUpgrade.
> > - SAMZA-1686: Set finite operation timeout when creating zkClient.
> > - SAMZA-1667: Skip storing configuration as a part of JobModel in
> > zookeeper> data nodes.
> > - SAMZA-1647: Fix NPE in JobModelExpired event handler.
> >
> > Eventhub
> > - SAMZA-1688: use per partition eventhubs client
> > - SAMZA-1676: miscellaneous fix and improvement for eventhubs system
> > - SAMZA-1656: EventHubSystemAdmin does not fetch metadata for valid
> > streams.>
> > host-affinity
> > - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> > requests> - SAMZA-1649: Improve host-aware allocation to account for
> strict
> > locality>
> > The complete list of changes are here:
> > https://issues.apache.org/jira/browse/SAMZA-1624?jql=
> project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.
> 14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs in the
> list have been completed and merged, and we
> > willl try to> get the remaining completed before 0.14.1 release.
> >
> > Here's what I propose:
> > - Cut the 0.14.1 release branch
> > - Target a release sometime next week (5/7 - 5/11).
> > Thoughts?
> >
> > Xinyu
>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: [Discuss] Samza 0.14.1 release
Posted by Tom Davis <to...@recursivedream.com>.
These all look like excellent changes to me, I'd be happy to have more
stability in Standalone!
Any timeline on getting Kafka 0.11 upgrade in? It seems to have been
done a while, but waiting for 0.15 I presume.
Best,
Tom
On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote:
> Hi, All,
>
> We have been adding many improvements and critical bug fixes in
> the areas> of Samza sql, standalone, eventhub system consumer and host-
> affinity since> 0.14.0 release. The changes should warrant a new minor release:
>
> SQL
> - SAMZA-1681: Samza-sql: Add support for handling older record schema> versions in AvroRelConverter
> - SAMZA-1671: SamzaSQL: add insert into table support
> - SAMZA-1651: Samza-sql: Implement GROUP BY SQL operator
>
> Standalone
> - SAMZA-1689: Add validations before state transitions in
> ZkBarrierForVersionUpgrade.
> - SAMZA-1686: Set finite operation timeout when creating zkClient.
> - SAMZA-1667: Skip storing configuration as a part of JobModel in
> zookeeper> data nodes.
> - SAMZA-1647: Fix NPE in JobModelExpired event handler.
>
> Eventhub
> - SAMZA-1688: use per partition eventhubs client
> - SAMZA-1676: miscellaneous fix and improvement for eventhubs system
> - SAMZA-1656: EventHubSystemAdmin does not fetch metadata for valid
> streams.>
> host-affinity
> - SAMZA-1687: Prioritize preferred host requests over ANY-HOST
> requests> - SAMZA-1649: Improve host-aware allocation to account for strict
> locality>
> The complete list of changes are here:
> https://issues.apache.org/jira/browse/SAMZA-1624?jql=project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.14.1%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC.> Most JIRAs in the list have been completed and merged, and we
> willl try to> get the remaining completed before 0.14.1 release.
>
> Here's what I propose:
> - Cut the 0.14.1 release branch
> - Target a release sometime next week (5/7 - 5/11).
> Thoughts?
>
> Xinyu