question about approach to take

[Ted Dunning <te...@gmail.com>]

There is a JIRA live for the network resilience feature that I mentioned
previously.

The design document
<https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing>
(also
copied into the JIRA) has essentially converged except for two points.

These include:

1) Artem Chernatsky has pointed out an opportunity to factor our port sets
in the configuration syntax as well as an interesting interaction with the
existing behavior where the current servers already listen to the specified
ports on all NICs. This semantics of this interaction between configuration
options need to be specified rigorously, but this doesn't appear to impact
code complexity much, nor introduce any real difficulties.

2) Alex Shraer seems to feel that there is a strong interaction between this
issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a proposed
refactorization of the configuration file syntax (mentioned in a comment in
3166, but apparently doesn't have an independent issue). In particular, he
seems to think that the syntax refactorization is a blocker for the network
resilience. My own feeling is that there is some interaction, but there is
no strong ordering between the two issues if the implementors of this issue
are willing to commit to supporting any consensus syntax change that is
adopted. Essentially, there can be an additional issue filed which is
blocked by both the syntax change issue and 3188 (network resilience) to
support any new syntax. The work for 3188 needs to support the old syntax
in any case so that we can backport changes to 3.4.

Other open issues that are affected by configuration syntax change include
2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
<https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
<https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
<https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these has
any serious impact other than the fact that configuration needs to be
abstracted as part of any change. Some appear to be quite old and may have
already been solved or made moot.

My own feeling is that pushing for this issue (3188) to include a change to
the configuration syntax as well as the core network resilience feature
proposed is an unacceptable increase in scope. I have filed a new tracking
issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
capturing the intended rework after a change in configuration syntax, but I
can't find anywhere that the configuration change is captured in a issue to
add the dependency.

I also see no particular way that configuration syntax change (as desirable
as it might be) blocks this feature.

I would love to hear other opinions.


I, myself, think that the "support resilience under new syntax when
available" approach

Re: question about approach to take

[Enrico Olivelli <eo...@gmail.com>]

Il giorno mar 13 nov 2018 alle ore 16:47 Alexander Shraer
<sh...@gmail.com> ha scritto:
>
> I also wanted to get other's views on this.
>
> My opinion is that the current server configuration format
> (server.x=ip:port:port:role;ip:port) has run its course. There are multiple
> proposals for additions/changes to the server configuration,
> that would be simplified from having a more extensible format, such as a
> json blob, as proposed by Brian Nixon here:
> https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> It is true that such an extension hasn't happened yet, however it may not
> be a good idea to continue adding individual features to the existing
> format instead of making this change.
>
> For longer than a year, maybe more, I've seen features pushed out to 3.6 to
> avoid destabilizing the 3.5 release. If we follow the same logic here, this
> would be a 3.6 feature, so compatibility with the old format doesn't seem
> very important.
>
> What do others think ?

Thank you Ted and Alexander for working on this important topic, I am
following your work.

Alexander
"so compatibility with the old format doesn't seem very important"

btw we must support compatibility, upgrade path from 3.5 must be as
smooth as possible

my 2c

Enrico

>
>
> Thanks,
> Alex
>
>
>
> On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com> wrote:
>
> > There is a JIRA live for the network resilience feature that I mentioned
> > previously.
> >
> > The design document
> > <
> > https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > >
> > (also
> > copied into the JIRA) has essentially converged except for two points.
> >
> > These include:
> >
> > 1) Artem Chernatsky has pointed out an opportunity to factor our port sets
> > in the configuration syntax as well as an interesting interaction with the
> > existing behavior where the current servers already listen to the specified
> > ports on all NICs. This semantics of this interaction between configuration
> > options need to be specified rigorously, but this doesn't appear to impact
> > code complexity much, nor introduce any real difficulties.
> >
> > 2) Alex Shraer seems to feel that there is a strong interaction between
> > this
> > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > proposed
> > refactorization of the configuration file syntax (mentioned in a comment in
> > 3166, but apparently doesn't have an independent issue). In particular, he
> > seems to think that the syntax refactorization is a blocker for the network
> > resilience. My own feeling is that there is some interaction, but there is
> > no strong ordering between the two issues if the implementors of this issue
> > are willing to commit to supporting any consensus syntax change that is
> > adopted. Essentially, there can be an additional issue filed which is
> > blocked by both the syntax change issue and 3188 (network resilience) to
> > support any new syntax. The work for 3188 needs to support the old syntax
> > in any case so that we can backport changes to 3.4.
> >
> > Other open issues that are affected by configuration syntax change include
> > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these has
> > any serious impact other than the fact that configuration needs to be
> > abstracted as part of any change. Some appear to be quite old and may have
> > already been solved or made moot.
> >
> > My own feeling is that pushing for this issue (3188) to include a change to
> > the configuration syntax as well as the core network resilience feature
> > proposed is an unacceptable increase in scope. I have filed a new tracking
> > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > capturing the intended rework after a change in configuration syntax, but I
> > can't find anywhere that the configuration change is captured in a issue to
> > add the dependency.
> >
> > I also see no particular way that configuration syntax change (as desirable
> > as it might be) blocks this feature.
> >
> > I would love to hear other opinions.
> >
> >
> > I, myself, think that the "support resilience under new syntax when
> > available" approach
> >

Re: question about approach to take

[timrc_us <ti...@yahoo.com>]

For 1), I posted this 3 or 4 years ago:
https://timmahnator.tumblr.com/post/115073121775/zoo-keepers-and-dock-workers

In the intervening time, with lots more "cloud" and "container" floating 
around,
I can only imagine the need has gotten greater.
And No, spinning up a DNS server inside and outside every container is 
not a good solution
to what should be a simple configuration option in ZK.

Thanks!

.timrc

Andor Molnar wrote:
> Hi Ted,
>
> Thanks for your contribution and the accurate design of this proposal. I
> have the following comments on it:
>
> 1)
> What's the need of the explicit support of multiple network interfaces? DNS
> names can resolve into multiple address and we could easily implement
> trying the multiple addresses to a single server in a round-robin fashion.
> Currently it's done by getByName() which always chooses the first one. What
> are the benefits of your implementation?
>
> 2)
> In terms of releases.
>
> The community already agreed on that 3.4 only accepts critical bug and
> security fixes. We could always start a vote on anything, but personally I
> doubt that such a significant change in the networking code could ever make
> it into 3.4. The reason why we're doing this is that 3.5 is just about to
> be released very soon, therefore we encourage new features and enhancements
> to be implemented 3.5 onwards.
>
> I believe it needs to get into master first. Once we have an insight on how
> significant the code change is exactly and the impact, we'll be able to
> talk about backporting it to 3.5.
>
> Don't worry about this too much. We don't want to wait another 4 years to
> release 3.6: with the current pace of patches from contributors (especially
> Facebook), we'll have another major (3.6) release very soon. Months rather
> than years.
>
> Regards,
> Andor
>
>
>
>
>
>
>
>
>
> On Tue, Nov 13, 2018 at 2:25 PM, Alexander Shraer <sh...@gmail.com> wrote:
>
>> This seems like a good feature for ZooKeeper to eventually have, but I
>> don't see why it must make 3.4 and 3.5 while other features are pushed out
>> to 3.6.
>> Like any other feature, it would be subject to a vote and isn't a
>> unilateral decision.
>>
>> On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com> wrote:
>>
>>> I am going to push this feature out sooner rather than later. That isn't
>> a
>>> question. I and my team are going to do the work. Others are very welcome
>>> to help and I am sure that there will be high value in getting reviews
>> from
>>> a wide group.
>>>
>>> But we are already working on the code. And we will be pushing a version
>>> into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
>>> improved configuration syntax. Better configuration is a great goal, but
>> it
>>> isn't OK to delay other work.
>>>
>>>
>>> On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
>>> wrote:
>>>
>>>> I also wanted to get other's views on this.
>>>>
>>>> My opinion is that the current server configuration format
>>>> (server.x=ip:port:port:role;ip:port) has run its course. There are
>>> multiple
>>>> proposals for additions/changes to the server configuration,
>>>> that would be simplified from having a more extensible format, such as
>> a
>>>> json blob, as proposed by Brian Nixon here:
>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3166
>>>> It is true that such an extension hasn't happened yet, however it may
>> not
>>>> be a good idea to continue adding individual features to the existing
>>>> format instead of making this change.
>>>>
>>>> For longer than a year, maybe more, I've seen features pushed out to
>> 3.6
>>> to
>>>> avoid destabilizing the 3.5 release. If we follow the same logic here,
>>> this
>>>> would be a 3.6 feature, so compatibility with the old format doesn't
>> seem
>>>> very important.
>>>>
>>>> What do others think ?
>>>>
>>>>
>>>> Thanks,
>>>> Alex
>>>>
>>>>
>>>>
>>>> On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
>>>> wrote:
>>>>
>>>>> There is a JIRA live for the network resilience feature that I
>>> mentioned
>>>>> previously.
>>>>>
>>>>> The design document
>>>>> <
>>>>>
>>> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_
>> kOQaJZ2GDo7j36fI/edit?usp=sharing
>>>>> (also
>>>>> copied into the JIRA) has essentially converged except for two
>> points.
>>>>> These include:
>>>>>
>>>>> 1) Artem Chernatsky has pointed out an opportunity to factor our port
>>>> sets
>>>>> in the configuration syntax as well as an interesting interaction
>> with
>>>> the
>>>>> existing behavior where the current servers already listen to the
>>>> specified
>>>>> ports on all NICs. This semantics of this interaction between
>>>> configuration
>>>>> options need to be specified rigorously, but this doesn't appear to
>>>> impact
>>>>> code complexity much, nor introduce any real difficulties.
>>>>>
>>>>> 2) Alex Shraer seems to feel that there is a strong interaction
>> between
>>>>> this
>>>>> issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
>>>>> proposed
>>>>> refactorization of the configuration file syntax (mentioned in a
>>> comment
>>>> in
>>>>> 3166, but apparently doesn't have an independent issue). In
>> particular,
>>>> he
>>>>> seems to think that the syntax refactorization is a blocker for the
>>>> network
>>>>> resilience. My own feeling is that there is some interaction, but
>> there
>>>> is
>>>>> no strong ordering between the two issues if the implementors of this
>>>> issue
>>>>> are willing to commit to supporting any consensus syntax change that
>> is
>>>>> adopted. Essentially, there can be an additional issue filed which is
>>>>> blocked by both the syntax change issue and 3188 (network resilience)
>>> to
>>>>> support any new syntax. The work for 3188 needs to support the old
>>> syntax
>>>>> in any case so that we can backport changes to 3.4.
>>>>>
>>>>> Other open issues that are affected by configuration syntax change
>>>> include
>>>>> 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
>>>>> <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
>>>>> <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
>>>>> <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of
>> these
>>>> has
>>>>> any serious impact other than the fact that configuration needs to be
>>>>> abstracted as part of any change. Some appear to be quite old and may
>>>> have
>>>>> already been solved or made moot.
>>>>>
>>>>> My own feeling is that pushing for this issue (3188) to include a
>>> change
>>>> to
>>>>> the configuration syntax as well as the core network resilience
>> feature
>>>>> proposed is an unacceptable increase in scope. I have filed a new
>>>> tracking
>>>>> issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
>>>>> capturing the intended rework after a change in configuration syntax,
>>>> but I
>>>>> can't find anywhere that the configuration change is captured in a
>>> issue
>>>> to
>>>>> add the dependency.
>>>>>
>>>>> I also see no particular way that configuration syntax change (as
>>>> desirable
>>>>> as it might be) blocks this feature.
>>>>>
>>>>> I would love to hear other opinions.
>>>>>
>>>>>
>>>>> I, myself, think that the "support resilience under new syntax when
>>>>> available" approach
>>>>>

Re: question about approach to take

[Michael Han <ha...@apache.org>]

I think there are two outstanding topics we need converge here.

First - does this work (ZOOKEEPER-3188) require a new configuration
subsystem (with new syntax, more powerful features etc)?
Second - what releases should ZOOKEEPER-3188 go in?

For first, technically ZOOKEEPER-3188 is not blocked by any work related to
the new configuration subsystem, as it seems totally possible to do this on
top of existing configuration format. However, I tend to agree with Alex
that having an improved configuration system that's more extensible would
be great, which will save time and effort later for the community as we
don't have to rewrite ZOOKEEPER-3188 to make it fit into the new subsystem.
Given how much folks want a new extensible system (to support things like
secure port, follower port that hosts observers, and many more), now it
seems a good time to do this first, though there is intrinsic dependency
between this and ZOOKEEPER-3188.

Second about release - I tend to agree with Alex and Andor that this
feature should not go to 3.4, because reasons Andor mentioned very well in
his reply. For 3.5, I have a mixed feeling because we are close to finally
reach stable release and having a dependency like this sounds not a good
idea - but I will leave the final decision to the release manager of the
upcoming stable 3.5 release. For 3.6, I think no one will object to land
this feature once it's available.

I also made some comments on ZOOKEEPER-3188 JIRA about high level design.

On Tue, Nov 13, 2018 at 3:17 PM Andor Molnar <an...@cloudera.com.invalid>
wrote:

> Hi Ted,
>
> Thanks for your contribution and the accurate design of this proposal. I
> have the following comments on it:
>
> 1)
> What's the need of the explicit support of multiple network interfaces? DNS
> names can resolve into multiple address and we could easily implement
> trying the multiple addresses to a single server in a round-robin fashion.
> Currently it's done by getByName() which always chooses the first one. What
> are the benefits of your implementation?
>
> 2)
> In terms of releases.
>
> The community already agreed on that 3.4 only accepts critical bug and
> security fixes. We could always start a vote on anything, but personally I
> doubt that such a significant change in the networking code could ever make
> it into 3.4. The reason why we're doing this is that 3.5 is just about to
> be released very soon, therefore we encourage new features and enhancements
> to be implemented 3.5 onwards.
>
> I believe it needs to get into master first. Once we have an insight on how
> significant the code change is exactly and the impact, we'll be able to
> talk about backporting it to 3.5.
>
> Don't worry about this too much. We don't want to wait another 4 years to
> release 3.6: with the current pace of patches from contributors (especially
> Facebook), we'll have another major (3.6) release very soon. Months rather
> than years.
>
> Regards,
> Andor
>
>
>
>
>
>
>
>
>
> On Tue, Nov 13, 2018 at 2:25 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > This seems like a good feature for ZooKeeper to eventually have, but I
> > don't see why it must make 3.4 and 3.5 while other features are pushed
> out
> > to 3.6.
> > Like any other feature, it would be subject to a vote and isn't a
> > unilateral decision.
> >
> > On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com>
> wrote:
> >
> > > I am going to push this feature out sooner rather than later. That
> isn't
> > a
> > > question. I and my team are going to do the work. Others are very
> welcome
> > > to help and I am sure that there will be high value in getting reviews
> > from
> > > a wide group.
> > >
> > > But we are already working on the code. And we will be pushing a
> version
> > > into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> > > improved configuration syntax. Better configuration is a great goal,
> but
> > it
> > > isn't OK to delay other work.
> > >
> > >
> > > On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> > > wrote:
> > >
> > > > I also wanted to get other's views on this.
> > > >
> > > > My opinion is that the current server configuration format
> > > > (server.x=ip:port:port:role;ip:port) has run its course. There are
> > > multiple
> > > > proposals for additions/changes to the server configuration,
> > > > that would be simplified from having a more extensible format, such
> as
> > a
> > > > json blob, as proposed by Brian Nixon here:
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > > > It is true that such an extension hasn't happened yet, however it may
> > not
> > > > be a good idea to continue adding individual features to the existing
> > > > format instead of making this change.
> > > >
> > > > For longer than a year, maybe more, I've seen features pushed out to
> > 3.6
> > > to
> > > > avoid destabilizing the 3.5 release. If we follow the same logic
> here,
> > > this
> > > > would be a 3.6 feature, so compatibility with the old format doesn't
> > seem
> > > > very important.
> > > >
> > > > What do others think ?
> > > >
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > >
> > > >
> > > > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > > > wrote:
> > > >
> > > > > There is a JIRA live for the network resilience feature that I
> > > mentioned
> > > > > previously.
> > > > >
> > > > > The design document
> > > > > <
> > > > >
> > > >
> > > https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_
> > kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > > > >
> > > > > (also
> > > > > copied into the JIRA) has essentially converged except for two
> > points.
> > > > >
> > > > > These include:
> > > > >
> > > > > 1) Artem Chernatsky has pointed out an opportunity to factor our
> port
> > > > sets
> > > > > in the configuration syntax as well as an interesting interaction
> > with
> > > > the
> > > > > existing behavior where the current servers already listen to the
> > > > specified
> > > > > ports on all NICs. This semantics of this interaction between
> > > > configuration
> > > > > options need to be specified rigorously, but this doesn't appear to
> > > > impact
> > > > > code complexity much, nor introduce any real difficulties.
> > > > >
> > > > > 2) Alex Shraer seems to feel that there is a strong interaction
> > between
> > > > > this
> > > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > > > proposed
> > > > > refactorization of the configuration file syntax (mentioned in a
> > > comment
> > > > in
> > > > > 3166, but apparently doesn't have an independent issue). In
> > particular,
> > > > he
> > > > > seems to think that the syntax refactorization is a blocker for the
> > > > network
> > > > > resilience. My own feeling is that there is some interaction, but
> > there
> > > > is
> > > > > no strong ordering between the two issues if the implementors of
> this
> > > > issue
> > > > > are willing to commit to supporting any consensus syntax change
> that
> > is
> > > > > adopted. Essentially, there can be an additional issue filed which
> is
> > > > > blocked by both the syntax change issue and 3188 (network
> resilience)
> > > to
> > > > > support any new syntax. The work for 3188 needs to support the old
> > > syntax
> > > > > in any case so that we can backport changes to 3.4.
> > > > >
> > > > > Other open issues that are affected by configuration syntax change
> > > > include
> > > > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of
> > these
> > > > has
> > > > > any serious impact other than the fact that configuration needs to
> be
> > > > > abstracted as part of any change. Some appear to be quite old and
> may
> > > > have
> > > > > already been solved or made moot.
> > > > >
> > > > > My own feeling is that pushing for this issue (3188) to include a
> > > change
> > > > to
> > > > > the configuration syntax as well as the core network resilience
> > feature
> > > > > proposed is an unacceptable increase in scope. I have filed a new
> > > > tracking
> > > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189>
> (3189)
> > > > > capturing the intended rework after a change in configuration
> syntax,
> > > > but I
> > > > > can't find anywhere that the configuration change is captured in a
> > > issue
> > > > to
> > > > > add the dependency.
> > > > >
> > > > > I also see no particular way that configuration syntax change (as
> > > > desirable
> > > > > as it might be) blocks this feature.
> > > > >
> > > > > I would love to hear other opinions.
> > > > >
> > > > >
> > > > > I, myself, think that the "support resilience under new syntax when
> > > > > available" approach
> > > > >
> > > >
> > >
> >
>

Re: question about approach to take

[Andor Molnar <an...@cloudera.com.INVALID>]

Hi Ted,

Thanks for your contribution and the accurate design of this proposal. I
have the following comments on it:

1)
What's the need of the explicit support of multiple network interfaces? DNS
names can resolve into multiple address and we could easily implement
trying the multiple addresses to a single server in a round-robin fashion.
Currently it's done by getByName() which always chooses the first one. What
are the benefits of your implementation?

2)
In terms of releases.

The community already agreed on that 3.4 only accepts critical bug and
security fixes. We could always start a vote on anything, but personally I
doubt that such a significant change in the networking code could ever make
it into 3.4. The reason why we're doing this is that 3.5 is just about to
be released very soon, therefore we encourage new features and enhancements
to be implemented 3.5 onwards.

I believe it needs to get into master first. Once we have an insight on how
significant the code change is exactly and the impact, we'll be able to
talk about backporting it to 3.5.

Don't worry about this too much. We don't want to wait another 4 years to
release 3.6: with the current pace of patches from contributors (especially
Facebook), we'll have another major (3.6) release very soon. Months rather
than years.

Regards,
Andor









On Tue, Nov 13, 2018 at 2:25 PM, Alexander Shraer <sh...@gmail.com> wrote:

> This seems like a good feature for ZooKeeper to eventually have, but I
> don't see why it must make 3.4 and 3.5 while other features are pushed out
> to 3.6.
> Like any other feature, it would be subject to a vote and isn't a
> unilateral decision.
>
> On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com> wrote:
>
> > I am going to push this feature out sooner rather than later. That isn't
> a
> > question. I and my team are going to do the work. Others are very welcome
> > to help and I am sure that there will be high value in getting reviews
> from
> > a wide group.
> >
> > But we are already working on the code. And we will be pushing a version
> > into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> > improved configuration syntax. Better configuration is a great goal, but
> it
> > isn't OK to delay other work.
> >
> >
> > On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > I also wanted to get other's views on this.
> > >
> > > My opinion is that the current server configuration format
> > > (server.x=ip:port:port:role;ip:port) has run its course. There are
> > multiple
> > > proposals for additions/changes to the server configuration,
> > > that would be simplified from having a more extensible format, such as
> a
> > > json blob, as proposed by Brian Nixon here:
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > > It is true that such an extension hasn't happened yet, however it may
> not
> > > be a good idea to continue adding individual features to the existing
> > > format instead of making this change.
> > >
> > > For longer than a year, maybe more, I've seen features pushed out to
> 3.6
> > to
> > > avoid destabilizing the 3.5 release. If we follow the same logic here,
> > this
> > > would be a 3.6 feature, so compatibility with the old format doesn't
> seem
> > > very important.
> > >
> > > What do others think ?
> > >
> > >
> > > Thanks,
> > > Alex
> > >
> > >
> > >
> > > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > > wrote:
> > >
> > > > There is a JIRA live for the network resilience feature that I
> > mentioned
> > > > previously.
> > > >
> > > > The design document
> > > > <
> > > >
> > >
> > https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_
> kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > > >
> > > > (also
> > > > copied into the JIRA) has essentially converged except for two
> points.
> > > >
> > > > These include:
> > > >
> > > > 1) Artem Chernatsky has pointed out an opportunity to factor our port
> > > sets
> > > > in the configuration syntax as well as an interesting interaction
> with
> > > the
> > > > existing behavior where the current servers already listen to the
> > > specified
> > > > ports on all NICs. This semantics of this interaction between
> > > configuration
> > > > options need to be specified rigorously, but this doesn't appear to
> > > impact
> > > > code complexity much, nor introduce any real difficulties.
> > > >
> > > > 2) Alex Shraer seems to feel that there is a strong interaction
> between
> > > > this
> > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > > proposed
> > > > refactorization of the configuration file syntax (mentioned in a
> > comment
> > > in
> > > > 3166, but apparently doesn't have an independent issue). In
> particular,
> > > he
> > > > seems to think that the syntax refactorization is a blocker for the
> > > network
> > > > resilience. My own feeling is that there is some interaction, but
> there
> > > is
> > > > no strong ordering between the two issues if the implementors of this
> > > issue
> > > > are willing to commit to supporting any consensus syntax change that
> is
> > > > adopted. Essentially, there can be an additional issue filed which is
> > > > blocked by both the syntax change issue and 3188 (network resilience)
> > to
> > > > support any new syntax. The work for 3188 needs to support the old
> > syntax
> > > > in any case so that we can backport changes to 3.4.
> > > >
> > > > Other open issues that are affected by configuration syntax change
> > > include
> > > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of
> these
> > > has
> > > > any serious impact other than the fact that configuration needs to be
> > > > abstracted as part of any change. Some appear to be quite old and may
> > > have
> > > > already been solved or made moot.
> > > >
> > > > My own feeling is that pushing for this issue (3188) to include a
> > change
> > > to
> > > > the configuration syntax as well as the core network resilience
> feature
> > > > proposed is an unacceptable increase in scope. I have filed a new
> > > tracking
> > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > > > capturing the intended rework after a change in configuration syntax,
> > > but I
> > > > can't find anywhere that the configuration change is captured in a
> > issue
> > > to
> > > > add the dependency.
> > > >
> > > > I also see no particular way that configuration syntax change (as
> > > desirable
> > > > as it might be) blocks this feature.
> > > >
> > > > I would love to hear other opinions.
> > > >
> > > >
> > > > I, myself, think that the "support resilience under new syntax when
> > > > available" approach
> > > >
> > >
> >
>

Re: question about approach to take

[Alexander Shraer <sh...@gmail.com>]

I provided technical objections, multiple times here and on the doc, based
on the fact that you propose a major extension that will further complicate
an already brittle configuration format, which multiple people are
similarly and concurrently also trying to extend in incompatible ways.
The validity of this objection is to be judged by other committers, and I
believe that Michael's email above confirms that my concern is valid.

Others have raised questions / concerns about the need for the feature, and
its interaction with multiple other features, which was not described in
the design doc.
Regarding the release - three committers have responded that this should
not go to 3.4, and that it is unlikely to get in 3.5 since it is a major
change and
will contribute to destabilizing the release.

Good luck,
Alex

On Tue, Nov 13, 2018 at 11:44 PM Ted Dunning <te...@gmail.com> wrote:

> The general rule at Apache is that if somebody has an itch to implement
> something, they do it.
>
> I have a strong need for this feature and will be implementing it. Doing so
> doesn't change what features get pushed.
>
> Regarding votes and such, commits are normally only subject to technical
> blocks. Absent a valid technical objection to using an existing
> configuration format, I plan to move forward with this. I am sympathetic to
> a desire for a new config and will even contribute to that effort, but I
> can't wait for it.
>
> On Tue, Nov 13, 2018, 22:25 Alexander Shraer <shralex@gmail.com wrote:
>
> > This seems like a good feature for ZooKeeper to eventually have, but I
> > don't see why it must make 3.4 and 3.5 while other features are pushed
> out
> > to 3.6.
> > Like any other feature, it would be subject to a vote and isn't a
> > unilateral decision.
> >
> > On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com>
> wrote:
> >
> > > I am going to push this feature out sooner rather than later. That
> isn't
> > a
> > > question. I and my team are going to do the work. Others are very
> welcome
> > > to help and I am sure that there will be high value in getting reviews
> > from
> > > a wide group.
> > >
> > > But we are already working on the code. And we will be pushing a
> version
> > > into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> > > improved configuration syntax. Better configuration is a great goal,
> but
> > it
> > > isn't OK to delay other work.
> > >
> > >
> > > On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> > > wrote:
> > >
> > > > I also wanted to get other's views on this.
> > > >
> > > > My opinion is that the current server configuration format
> > > > (server.x=ip:port:port:role;ip:port) has run its course. There are
> > > multiple
> > > > proposals for additions/changes to the server configuration,
> > > > that would be simplified from having a more extensible format, such
> as
> > a
> > > > json blob, as proposed by Brian Nixon here:
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > > > It is true that such an extension hasn't happened yet, however it may
> > not
> > > > be a good idea to continue adding individual features to the existing
> > > > format instead of making this change.
> > > >
> > > > For longer than a year, maybe more, I've seen features pushed out to
> > 3.6
> > > to
> > > > avoid destabilizing the 3.5 release. If we follow the same logic
> here,
> > > this
> > > > would be a 3.6 feature, so compatibility with the old format doesn't
> > seem
> > > > very important.
> > > >
> > > > What do others think ?
> > > >
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > >
> > > >
> > > > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > > > wrote:
> > > >
> > > > > There is a JIRA live for the network resilience feature that I
> > > mentioned
> > > > > previously.
> > > > >
> > > > > The design document
> > > > > <
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > > > >
> > > > > (also
> > > > > copied into the JIRA) has essentially converged except for two
> > points.
> > > > >
> > > > > These include:
> > > > >
> > > > > 1) Artem Chernatsky has pointed out an opportunity to factor our
> port
> > > > sets
> > > > > in the configuration syntax as well as an interesting interaction
> > with
> > > > the
> > > > > existing behavior where the current servers already listen to the
> > > > specified
> > > > > ports on all NICs. This semantics of this interaction between
> > > > configuration
> > > > > options need to be specified rigorously, but this doesn't appear to
> > > > impact
> > > > > code complexity much, nor introduce any real difficulties.
> > > > >
> > > > > 2) Alex Shraer seems to feel that there is a strong interaction
> > between
> > > > > this
> > > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > > > proposed
> > > > > refactorization of the configuration file syntax (mentioned in a
> > > comment
> > > > in
> > > > > 3166, but apparently doesn't have an independent issue). In
> > particular,
> > > > he
> > > > > seems to think that the syntax refactorization is a blocker for the
> > > > network
> > > > > resilience. My own feeling is that there is some interaction, but
> > there
> > > > is
> > > > > no strong ordering between the two issues if the implementors of
> this
> > > > issue
> > > > > are willing to commit to supporting any consensus syntax change
> that
> > is
> > > > > adopted. Essentially, there can be an additional issue filed which
> is
> > > > > blocked by both the syntax change issue and 3188 (network
> resilience)
> > > to
> > > > > support any new syntax. The work for 3188 needs to support the old
> > > syntax
> > > > > in any case so that we can backport changes to 3.4.
> > > > >
> > > > > Other open issues that are affected by configuration syntax change
> > > > include
> > > > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of
> > these
> > > > has
> > > > > any serious impact other than the fact that configuration needs to
> be
> > > > > abstracted as part of any change. Some appear to be quite old and
> may
> > > > have
> > > > > already been solved or made moot.
> > > > >
> > > > > My own feeling is that pushing for this issue (3188) to include a
> > > change
> > > > to
> > > > > the configuration syntax as well as the core network resilience
> > feature
> > > > > proposed is an unacceptable increase in scope. I have filed a new
> > > > tracking
> > > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189>
> (3189)
> > > > > capturing the intended rework after a change in configuration
> syntax,
> > > > but I
> > > > > can't find anywhere that the configuration change is captured in a
> > > issue
> > > > to
> > > > > add the dependency.
> > > > >
> > > > > I also see no particular way that configuration syntax change (as
> > > > desirable
> > > > > as it might be) blocks this feature.
> > > > >
> > > > > I would love to hear other opinions.
> > > > >
> > > > >
> > > > > I, myself, think that the "support resilience under new syntax when
> > > > > available" approach
> > > > >
> > > >
> > >
> >
>

Re: question about approach to take

[Ted Dunning <te...@gmail.com>]

The general rule at Apache is that if somebody has an itch to implement
something, they do it.

I have a strong need for this feature and will be implementing it. Doing so
doesn't change what features get pushed.

Regarding votes and such, commits are normally only subject to technical
blocks. Absent a valid technical objection to using an existing
configuration format, I plan to move forward with this. I am sympathetic to
a desire for a new config and will even contribute to that effort, but I
can't wait for it.

On Tue, Nov 13, 2018, 22:25 Alexander Shraer <shralex@gmail.com wrote:

> This seems like a good feature for ZooKeeper to eventually have, but I
> don't see why it must make 3.4 and 3.5 while other features are pushed out
> to 3.6.
> Like any other feature, it would be subject to a vote and isn't a
> unilateral decision.
>
> On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com> wrote:
>
> > I am going to push this feature out sooner rather than later. That isn't
> a
> > question. I and my team are going to do the work. Others are very welcome
> > to help and I am sure that there will be high value in getting reviews
> from
> > a wide group.
> >
> > But we are already working on the code. And we will be pushing a version
> > into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> > improved configuration syntax. Better configuration is a great goal, but
> it
> > isn't OK to delay other work.
> >
> >
> > On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > I also wanted to get other's views on this.
> > >
> > > My opinion is that the current server configuration format
> > > (server.x=ip:port:port:role;ip:port) has run its course. There are
> > multiple
> > > proposals for additions/changes to the server configuration,
> > > that would be simplified from having a more extensible format, such as
> a
> > > json blob, as proposed by Brian Nixon here:
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > > It is true that such an extension hasn't happened yet, however it may
> not
> > > be a good idea to continue adding individual features to the existing
> > > format instead of making this change.
> > >
> > > For longer than a year, maybe more, I've seen features pushed out to
> 3.6
> > to
> > > avoid destabilizing the 3.5 release. If we follow the same logic here,
> > this
> > > would be a 3.6 feature, so compatibility with the old format doesn't
> seem
> > > very important.
> > >
> > > What do others think ?
> > >
> > >
> > > Thanks,
> > > Alex
> > >
> > >
> > >
> > > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > > wrote:
> > >
> > > > There is a JIRA live for the network resilience feature that I
> > mentioned
> > > > previously.
> > > >
> > > > The design document
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > > >
> > > > (also
> > > > copied into the JIRA) has essentially converged except for two
> points.
> > > >
> > > > These include:
> > > >
> > > > 1) Artem Chernatsky has pointed out an opportunity to factor our port
> > > sets
> > > > in the configuration syntax as well as an interesting interaction
> with
> > > the
> > > > existing behavior where the current servers already listen to the
> > > specified
> > > > ports on all NICs. This semantics of this interaction between
> > > configuration
> > > > options need to be specified rigorously, but this doesn't appear to
> > > impact
> > > > code complexity much, nor introduce any real difficulties.
> > > >
> > > > 2) Alex Shraer seems to feel that there is a strong interaction
> between
> > > > this
> > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > > proposed
> > > > refactorization of the configuration file syntax (mentioned in a
> > comment
> > > in
> > > > 3166, but apparently doesn't have an independent issue). In
> particular,
> > > he
> > > > seems to think that the syntax refactorization is a blocker for the
> > > network
> > > > resilience. My own feeling is that there is some interaction, but
> there
> > > is
> > > > no strong ordering between the two issues if the implementors of this
> > > issue
> > > > are willing to commit to supporting any consensus syntax change that
> is
> > > > adopted. Essentially, there can be an additional issue filed which is
> > > > blocked by both the syntax change issue and 3188 (network resilience)
> > to
> > > > support any new syntax. The work for 3188 needs to support the old
> > syntax
> > > > in any case so that we can backport changes to 3.4.
> > > >
> > > > Other open issues that are affected by configuration syntax change
> > > include
> > > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of
> these
> > > has
> > > > any serious impact other than the fact that configuration needs to be
> > > > abstracted as part of any change. Some appear to be quite old and may
> > > have
> > > > already been solved or made moot.
> > > >
> > > > My own feeling is that pushing for this issue (3188) to include a
> > change
> > > to
> > > > the configuration syntax as well as the core network resilience
> feature
> > > > proposed is an unacceptable increase in scope. I have filed a new
> > > tracking
> > > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > > > capturing the intended rework after a change in configuration syntax,
> > > but I
> > > > can't find anywhere that the configuration change is captured in a
> > issue
> > > to
> > > > add the dependency.
> > > >
> > > > I also see no particular way that configuration syntax change (as
> > > desirable
> > > > as it might be) blocks this feature.
> > > >
> > > > I would love to hear other opinions.
> > > >
> > > >
> > > > I, myself, think that the "support resilience under new syntax when
> > > > available" approach
> > > >
> > >
> >
>

Re: question about approach to take

[Alexander Shraer <sh...@gmail.com>]

This seems like a good feature for ZooKeeper to eventually have, but I
don't see why it must make 3.4 and 3.5 while other features are pushed out
to 3.6.
Like any other feature, it would be subject to a vote and isn't a
unilateral decision.

On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com> wrote:

> I am going to push this feature out sooner rather than later. That isn't a
> question. I and my team are going to do the work. Others are very welcome
> to help and I am sure that there will be high value in getting reviews from
> a wide group.
>
> But we are already working on the code. And we will be pushing a version
> into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> improved configuration syntax. Better configuration is a great goal, but it
> isn't OK to delay other work.
>
>
> On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > I also wanted to get other's views on this.
> >
> > My opinion is that the current server configuration format
> > (server.x=ip:port:port:role;ip:port) has run its course. There are
> multiple
> > proposals for additions/changes to the server configuration,
> > that would be simplified from having a more extensible format, such as a
> > json blob, as proposed by Brian Nixon here:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > It is true that such an extension hasn't happened yet, however it may not
> > be a good idea to continue adding individual features to the existing
> > format instead of making this change.
> >
> > For longer than a year, maybe more, I've seen features pushed out to 3.6
> to
> > avoid destabilizing the 3.5 release. If we follow the same logic here,
> this
> > would be a 3.6 feature, so compatibility with the old format doesn't seem
> > very important.
> >
> > What do others think ?
> >
> >
> > Thanks,
> > Alex
> >
> >
> >
> > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > There is a JIRA live for the network resilience feature that I
> mentioned
> > > previously.
> > >
> > > The design document
> > > <
> > >
> >
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > >
> > > (also
> > > copied into the JIRA) has essentially converged except for two points.
> > >
> > > These include:
> > >
> > > 1) Artem Chernatsky has pointed out an opportunity to factor our port
> > sets
> > > in the configuration syntax as well as an interesting interaction with
> > the
> > > existing behavior where the current servers already listen to the
> > specified
> > > ports on all NICs. This semantics of this interaction between
> > configuration
> > > options need to be specified rigorously, but this doesn't appear to
> > impact
> > > code complexity much, nor introduce any real difficulties.
> > >
> > > 2) Alex Shraer seems to feel that there is a strong interaction between
> > > this
> > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > proposed
> > > refactorization of the configuration file syntax (mentioned in a
> comment
> > in
> > > 3166, but apparently doesn't have an independent issue). In particular,
> > he
> > > seems to think that the syntax refactorization is a blocker for the
> > network
> > > resilience. My own feeling is that there is some interaction, but there
> > is
> > > no strong ordering between the two issues if the implementors of this
> > issue
> > > are willing to commit to supporting any consensus syntax change that is
> > > adopted. Essentially, there can be an additional issue filed which is
> > > blocked by both the syntax change issue and 3188 (network resilience)
> to
> > > support any new syntax. The work for 3188 needs to support the old
> syntax
> > > in any case so that we can backport changes to 3.4.
> > >
> > > Other open issues that are affected by configuration syntax change
> > include
> > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these
> > has
> > > any serious impact other than the fact that configuration needs to be
> > > abstracted as part of any change. Some appear to be quite old and may
> > have
> > > already been solved or made moot.
> > >
> > > My own feeling is that pushing for this issue (3188) to include a
> change
> > to
> > > the configuration syntax as well as the core network resilience feature
> > > proposed is an unacceptable increase in scope. I have filed a new
> > tracking
> > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > > capturing the intended rework after a change in configuration syntax,
> > but I
> > > can't find anywhere that the configuration change is captured in a
> issue
> > to
> > > add the dependency.
> > >
> > > I also see no particular way that configuration syntax change (as
> > desirable
> > > as it might be) blocks this feature.
> > >
> > > I would love to hear other opinions.
> > >
> > >
> > > I, myself, think that the "support resilience under new syntax when
> > > available" approach
> > >
> >
>

Re: question about approach to take

[Patrick Hunt <ph...@apache.org>]

On Tue, Nov 13, 2018 at 1:59 PM Ted Dunning <te...@gmail.com> wrote:

> I am going to push this feature out sooner rather than later. That isn't a
> question. I and my team are going to do the work. Others are very welcome
> to help and I am sure that there will be high value in getting reviews from
> a wide group.
>
> But we are already working on the code. And we will be pushing a version
> into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
> improved configuration syntax. Better configuration is a great goal, but it
> isn't OK to delay other work.
>
>
We've had a long established policy of only making fixes in the fix
releases, and as I've stated before we should stop making
feature/improvement changes in 3.5 so that it can stabilize. Active
development (e.g. the great patches from Alex and recently Fangmin and
others) have been landing in trunk/master. If the community can reach
consensus on the proposed changes it would make sense to me to include in
3.6 or later.

Regards,

Patrick


>
> On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > I also wanted to get other's views on this.
> >
> > My opinion is that the current server configuration format
> > (server.x=ip:port:port:role;ip:port) has run its course. There are
> multiple
> > proposals for additions/changes to the server configuration,
> > that would be simplified from having a more extensible format, such as a
> > json blob, as proposed by Brian Nixon here:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> > It is true that such an extension hasn't happened yet, however it may not
> > be a good idea to continue adding individual features to the existing
> > format instead of making this change.
> >
> > For longer than a year, maybe more, I've seen features pushed out to 3.6
> to
> > avoid destabilizing the 3.5 release. If we follow the same logic here,
> this
> > would be a 3.6 feature, so compatibility with the old format doesn't seem
> > very important.
> >
> > What do others think ?
> >
> >
> > Thanks,
> > Alex
> >
> >
> >
> > On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > There is a JIRA live for the network resilience feature that I
> mentioned
> > > previously.
> > >
> > > The design document
> > > <
> > >
> >
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > > >
> > > (also
> > > copied into the JIRA) has essentially converged except for two points.
> > >
> > > These include:
> > >
> > > 1) Artem Chernatsky has pointed out an opportunity to factor our port
> > sets
> > > in the configuration syntax as well as an interesting interaction with
> > the
> > > existing behavior where the current servers already listen to the
> > specified
> > > ports on all NICs. This semantics of this interaction between
> > configuration
> > > options need to be specified rigorously, but this doesn't appear to
> > impact
> > > code complexity much, nor introduce any real difficulties.
> > >
> > > 2) Alex Shraer seems to feel that there is a strong interaction between
> > > this
> > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > > proposed
> > > refactorization of the configuration file syntax (mentioned in a
> comment
> > in
> > > 3166, but apparently doesn't have an independent issue). In particular,
> > he
> > > seems to think that the syntax refactorization is a blocker for the
> > network
> > > resilience. My own feeling is that there is some interaction, but there
> > is
> > > no strong ordering between the two issues if the implementors of this
> > issue
> > > are willing to commit to supporting any consensus syntax change that is
> > > adopted. Essentially, there can be an additional issue filed which is
> > > blocked by both the syntax change issue and 3188 (network resilience)
> to
> > > support any new syntax. The work for 3188 needs to support the old
> syntax
> > > in any case so that we can backport changes to 3.4.
> > >
> > > Other open issues that are affected by configuration syntax change
> > include
> > > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these
> > has
> > > any serious impact other than the fact that configuration needs to be
> > > abstracted as part of any change. Some appear to be quite old and may
> > have
> > > already been solved or made moot.
> > >
> > > My own feeling is that pushing for this issue (3188) to include a
> change
> > to
> > > the configuration syntax as well as the core network resilience feature
> > > proposed is an unacceptable increase in scope. I have filed a new
> > tracking
> > > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > > capturing the intended rework after a change in configuration syntax,
> > but I
> > > can't find anywhere that the configuration change is captured in a
> issue
> > to
> > > add the dependency.
> > >
> > > I also see no particular way that configuration syntax change (as
> > desirable
> > > as it might be) blocks this feature.
> > >
> > > I would love to hear other opinions.
> > >
> > >
> > > I, myself, think that the "support resilience under new syntax when
> > > available" approach
> > >
> >
>

Re: question about approach to take

[Ted Dunning <te...@gmail.com>]

I am going to push this feature out sooner rather than later. That isn't a
question. I and my team are going to do the work. Others are very welcome
to help and I am sure that there will be high value in getting reviews from
a wide group.

But we are already working on the code. And we will be pushing a version
into both 3.4 and 3.5. I think that 3.6-ish is a great target for an
improved configuration syntax. Better configuration is a great goal, but it
isn't OK to delay other work.


On Tue, Nov 13, 2018 at 3:47 PM Alexander Shraer <sh...@gmail.com> wrote:

> I also wanted to get other's views on this.
>
> My opinion is that the current server configuration format
> (server.x=ip:port:port:role;ip:port) has run its course. There are multiple
> proposals for additions/changes to the server configuration,
> that would be simplified from having a more extensible format, such as a
> json blob, as proposed by Brian Nixon here:
> https://issues.apache.org/jira/browse/ZOOKEEPER-3166
> It is true that such an extension hasn't happened yet, however it may not
> be a good idea to continue adding individual features to the existing
> format instead of making this change.
>
> For longer than a year, maybe more, I've seen features pushed out to 3.6 to
> avoid destabilizing the 3.5 release. If we follow the same logic here, this
> would be a 3.6 feature, so compatibility with the old format doesn't seem
> very important.
>
> What do others think ?
>
>
> Thanks,
> Alex
>
>
>
> On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com>
> wrote:
>
> > There is a JIRA live for the network resilience feature that I mentioned
> > previously.
> >
> > The design document
> > <
> >
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> > >
> > (also
> > copied into the JIRA) has essentially converged except for two points.
> >
> > These include:
> >
> > 1) Artem Chernatsky has pointed out an opportunity to factor our port
> sets
> > in the configuration syntax as well as an interesting interaction with
> the
> > existing behavior where the current servers already listen to the
> specified
> > ports on all NICs. This semantics of this interaction between
> configuration
> > options need to be specified rigorously, but this doesn't appear to
> impact
> > code complexity much, nor introduce any real difficulties.
> >
> > 2) Alex Shraer seems to feel that there is a strong interaction between
> > this
> > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> > proposed
> > refactorization of the configuration file syntax (mentioned in a comment
> in
> > 3166, but apparently doesn't have an independent issue). In particular,
> he
> > seems to think that the syntax refactorization is a blocker for the
> network
> > resilience. My own feeling is that there is some interaction, but there
> is
> > no strong ordering between the two issues if the implementors of this
> issue
> > are willing to commit to supporting any consensus syntax change that is
> > adopted. Essentially, there can be an additional issue filed which is
> > blocked by both the syntax change issue and 3188 (network resilience) to
> > support any new syntax. The work for 3188 needs to support the old syntax
> > in any case so that we can backport changes to 3.4.
> >
> > Other open issues that are affected by configuration syntax change
> include
> > 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> > <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these
> has
> > any serious impact other than the fact that configuration needs to be
> > abstracted as part of any change. Some appear to be quite old and may
> have
> > already been solved or made moot.
> >
> > My own feeling is that pushing for this issue (3188) to include a change
> to
> > the configuration syntax as well as the core network resilience feature
> > proposed is an unacceptable increase in scope. I have filed a new
> tracking
> > issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> > capturing the intended rework after a change in configuration syntax,
> but I
> > can't find anywhere that the configuration change is captured in a issue
> to
> > add the dependency.
> >
> > I also see no particular way that configuration syntax change (as
> desirable
> > as it might be) blocks this feature.
> >
> > I would love to hear other opinions.
> >
> >
> > I, myself, think that the "support resilience under new syntax when
> > available" approach
> >
>

Re: question about approach to take

[Alexander Shraer <sh...@gmail.com>]

I also wanted to get other's views on this.

My opinion is that the current server configuration format
(server.x=ip:port:port:role;ip:port) has run its course. There are multiple
proposals for additions/changes to the server configuration,
that would be simplified from having a more extensible format, such as a
json blob, as proposed by Brian Nixon here:
https://issues.apache.org/jira/browse/ZOOKEEPER-3166
It is true that such an extension hasn't happened yet, however it may not
be a good idea to continue adding individual features to the existing
format instead of making this change.

For longer than a year, maybe more, I've seen features pushed out to 3.6 to
avoid destabilizing the 3.5 release. If we follow the same logic here, this
would be a 3.6 feature, so compatibility with the old format doesn't seem
very important.

What do others think ?


Thanks,
Alex



On Mon, Nov 12, 2018 at 11:47 PM Ted Dunning <te...@gmail.com> wrote:

> There is a JIRA live for the network resilience feature that I mentioned
> previously.
>
> The design document
> <
> https://docs.google.com/document/d/1iGVwxeHp57qogwfdodCh9b32P2_kOQaJZ2GDo7j36fI/edit?usp=sharing
> >
> (also
> copied into the JIRA) has essentially converged except for two points.
>
> These include:
>
> 1) Artem Chernatsky has pointed out an opportunity to factor our port sets
> in the configuration syntax as well as an interesting interaction with the
> existing behavior where the current servers already listen to the specified
> ports on all NICs. This semantics of this interaction between configuration
> options need to be specified rigorously, but this doesn't appear to impact
> code complexity much, nor introduce any real difficulties.
>
> 2) Alex Shraer seems to feel that there is a strong interaction between
> this
> issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3188> and a
> proposed
> refactorization of the configuration file syntax (mentioned in a comment in
> 3166, but apparently doesn't have an independent issue). In particular, he
> seems to think that the syntax refactorization is a blocker for the network
> resilience. My own feeling is that there is some interaction, but there is
> no strong ordering between the two issues if the implementors of this issue
> are willing to commit to supporting any consensus syntax change that is
> adopted. Essentially, there can be an additional issue filed which is
> blocked by both the syntax change issue and 3188 (network resilience) to
> support any new syntax. The work for 3188 needs to support the old syntax
> in any case so that we can backport changes to 3.4.
>
> Other open issues that are affected by configuration syntax change include
> 2534 <https://issues.apache.org/jira/browse/ZOOKEEPER-2534>, 2531
> <https://issues.apache.org/jira/browse/ZOOKEEPER-2531>, 195
> <https://issues.apache.org/jira/browse/ZOOKEEPER-195>, and 2225
> <https://issues.apache.org/jira/browse/ZOOKEEPER-2225>. None of these has
> any serious impact other than the fact that configuration needs to be
> abstracted as part of any change. Some appear to be quite old and may have
> already been solved or made moot.
>
> My own feeling is that pushing for this issue (3188) to include a change to
> the configuration syntax as well as the core network resilience feature
> proposed is an unacceptable increase in scope. I have filed a new tracking
> issue <https://issues.apache.org/jira/browse/ZOOKEEPER-3189> (3189)
> capturing the intended rework after a change in configuration syntax, but I
> can't find anywhere that the configuration change is captured in a issue to
> add the dependency.
>
> I also see no particular way that configuration syntax change (as desirable
> as it might be) blocks this feature.
>
> I would love to hear other opinions.
>
>
> I, myself, think that the "support resilience under new syntax when
> available" approach
>