You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Bruce Schuchardt <bs...@pivotal.io> on 2016/01/07 22:18:23 UTC
enabling network partition detection by default
Another thing that's been discussed for a long time is turning on
network-partition-detection by default. It is a major problem for
someone if a partition occurs and they are using persistence. The
disk-stores on all but one of the partitions have to be deleted and revoked.
Re: enabling network partition detection by default
Posted by Dan Smith <ds...@pivotal.io>.
++1
On Thu, Jan 7, 2016 at 2:12 PM, Swapnil Bawaskar <sb...@pivotal.io>
wrote:
> This will have to be phased when we do it, as it will not work with rolling
> upgrades.
>
All the more reason to make the change for geode 1.0.0 ;)
-Dan
Re: enabling network partition detection by default
Posted by William Markito <wm...@pivotal.io>.
+1 and I would also say that we should use that to review other default
settings as well, such as conserve-sockets and etc...
On Thu, Jan 7, 2016 at 2:17 PM, Kirk Lund <kl...@pivotal.io> wrote:
> This 1st geode release (and any alpha flavors of it) will not support
> rolling upgrade because of the new jgroups . So now would be a good time to
> change defaults such as this.
>
> -Kirk
>
>
> On Thu, Jan 7, 2016 at 2:12 PM, Swapnil Bawaskar <sb...@pivotal.io>
> wrote:
>
> > This will have to be phased when we do it, as it will not work with
> rolling
> > upgrades.
> >
> > During a rolling upgrade, an existing member will not accept the new
> member
> > because the "network-partition-detection" property will not have the same
> > value.
> >
> > On Thu, Jan 7, 2016 at 1:31 PM, Michael Stolz <ms...@pivotal.io> wrote:
> >
> > > I never knew that. Yikes. Given that just using PDX causes the use of
> > disk
> > > stores this seems really important.
> > >
> > > --
> > > Mike Stolz
> > > Principal Engineer, GemFire Product Manager
> > > Mobile: 631-835-4771
> > >
> > > On Thu, Jan 7, 2016 at 4:18 PM, Bruce Schuchardt <
> bschuchardt@pivotal.io
> > >
> > > wrote:
> > >
> > > > Another thing that's been discussed for a long time is turning on
> > > > network-partition-detection by default. It is a major problem for
> > > someone
> > > > if a partition occurs and they are using persistence. The
> disk-stores
> > on
> > > > all but one of the partitions have to be deleted and revoked.
> > > >
> > >
> >
>
--
William Markito Oliveira
-- For questions about Apache Geode, please write to
*dev@geode.incubator.apache.org
<de...@geode.incubator.apache.org>*
Re: enabling network partition detection by default
Posted by Kirk Lund <kl...@pivotal.io>.
This 1st geode release (and any alpha flavors of it) will not support
rolling upgrade because of the new jgroups . So now would be a good time to
change defaults such as this.
-Kirk
On Thu, Jan 7, 2016 at 2:12 PM, Swapnil Bawaskar <sb...@pivotal.io>
wrote:
> This will have to be phased when we do it, as it will not work with rolling
> upgrades.
>
> During a rolling upgrade, an existing member will not accept the new member
> because the "network-partition-detection" property will not have the same
> value.
>
> On Thu, Jan 7, 2016 at 1:31 PM, Michael Stolz <ms...@pivotal.io> wrote:
>
> > I never knew that. Yikes. Given that just using PDX causes the use of
> disk
> > stores this seems really important.
> >
> > --
> > Mike Stolz
> > Principal Engineer, GemFire Product Manager
> > Mobile: 631-835-4771
> >
> > On Thu, Jan 7, 2016 at 4:18 PM, Bruce Schuchardt <bschuchardt@pivotal.io
> >
> > wrote:
> >
> > > Another thing that's been discussed for a long time is turning on
> > > network-partition-detection by default. It is a major problem for
> > someone
> > > if a partition occurs and they are using persistence. The disk-stores
> on
> > > all but one of the partitions have to be deleted and revoked.
> > >
> >
>
Re: enabling network partition detection by default
Posted by Swapnil Bawaskar <sb...@pivotal.io>.
This will have to be phased when we do it, as it will not work with rolling
upgrades.
During a rolling upgrade, an existing member will not accept the new member
because the "network-partition-detection" property will not have the same
value.
On Thu, Jan 7, 2016 at 1:31 PM, Michael Stolz <ms...@pivotal.io> wrote:
> I never knew that. Yikes. Given that just using PDX causes the use of disk
> stores this seems really important.
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Thu, Jan 7, 2016 at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
> wrote:
>
> > Another thing that's been discussed for a long time is turning on
> > network-partition-detection by default. It is a major problem for
> someone
> > if a partition occurs and they are using persistence. The disk-stores on
> > all but one of the partitions have to be deleted and revoked.
> >
>
Re: enabling network partition detection by default
Posted by Darrel Schneider <ds...@pivotal.io>.
My understanding is that by default PDX does not use a disk store. You need
to set "pdx-persistent" to true.
The default is false. But if you have any persistence in a member and use
pdx you will get an exception if you don't set pdx-persistent to true.
On Thu, Jan 7, 2016 at 1:31 PM, Michael Stolz <ms...@pivotal.io> wrote:
> I never knew that. Yikes. Given that just using PDX causes the use of disk
> stores this seems really important.
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Thu, Jan 7, 2016 at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
> wrote:
>
> > Another thing that's been discussed for a long time is turning on
> > network-partition-detection by default. It is a major problem for
> someone
> > if a partition occurs and they are using persistence. The disk-stores on
> > all but one of the partitions have to be deleted and revoked.
> >
>
Re: enabling network partition detection by default
Posted by Michael Stolz <ms...@pivotal.io>.
I never knew that. Yikes. Given that just using PDX causes the use of disk
stores this seems really important.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Thu, Jan 7, 2016 at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
wrote:
> Another thing that's been discussed for a long time is turning on
> network-partition-detection by default. It is a major problem for someone
> if a partition occurs and they are using persistence. The disk-stores on
> all but one of the partitions have to be deleted and revoked.
>
Re: enabling network partition detection by default
Posted by Bruce Schuchardt <bs...@pivotal.io>.
Yes, it should definitely be in the release notes. If folks are okay
with the legacy_default.properties file Jianxia can create that.
Le 1/8/2016 11:57 AM, Michael Stolz a écrit :
> I don't think I got my point across correctly.
>
> If we change defaults we need to document that, of course, and not just in
> our 1000+ page manual because that will not be read soon enough to avoid
> problems from the changed defaults. We definitely need that to appear in
> the release notes.
>
> I think we should create a "legacy_default.properties" file that contains
> the old default settings for any defaults we are going to change and make
> that available to legacy users so that they KNOW what the defaults WERE,
> and can use some or all of that file to augment their current
> gemfire.properties file to put the defaults back the way they have been
> running up until now.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Fri, Jan 8, 2016 at 2:38 PM, Bruce Schuchardt <bs...@pivotal.io>
> wrote:
>
>> jchen21 (Jianxia) has volunteered to do this work and is looking into what
>> needs to be done.
>>
>>
>> Le 1/7/2016 1:18 PM, Bruce Schuchardt a écrit :
>>
>>> Another thing that's been discussed for a long time is turning on
>>> network-partition-detection by default. It is a major problem for someone
>>> if a partition occurs and they are using persistence. The disk-stores on
>>> all but one of the partitions have to be deleted and revoked.
>>>
>>
Re: enabling network partition detection by default
Posted by Michael Stolz <ms...@pivotal.io>.
I don't think I got my point across correctly.
If we change defaults we need to document that, of course, and not just in
our 1000+ page manual because that will not be read soon enough to avoid
problems from the changed defaults. We definitely need that to appear in
the release notes.
I think we should create a "legacy_default.properties" file that contains
the old default settings for any defaults we are going to change and make
that available to legacy users so that they KNOW what the defaults WERE,
and can use some or all of that file to augment their current
gemfire.properties file to put the defaults back the way they have been
running up until now.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Fri, Jan 8, 2016 at 2:38 PM, Bruce Schuchardt <bs...@pivotal.io>
wrote:
> jchen21 (Jianxia) has volunteered to do this work and is looking into what
> needs to be done.
>
>
> Le 1/7/2016 1:18 PM, Bruce Schuchardt a écrit :
>
>> Another thing that's been discussed for a long time is turning on
>> network-partition-detection by default. It is a major problem for someone
>> if a partition occurs and they are using persistence. The disk-stores on
>> all but one of the partitions have to be deleted and revoked.
>>
>
>
Re: enabling network partition detection by default
Posted by Bruce Schuchardt <bs...@pivotal.io>.
jchen21 (Jianxia) has volunteered to do this work and is looking into
what needs to be done.
Le 1/7/2016 1:18 PM, Bruce Schuchardt a écrit :
> Another thing that's been discussed for a long time is turning on
> network-partition-detection by default. It is a major problem for
> someone if a partition occurs and they are using persistence. The
> disk-stores on all but one of the partitions have to be deleted and
> revoked.
Re: enabling network partition detection by default
Posted by Kirk Lund <kl...@pivotal.io>.
Having a second set of properties is just one more thing that a user has to
read but will ultimately not look at or not know about. That's why changing
the defaults would be valuable. Either the existing users (who have more
experience) need to be made aware of the change in defaults or the new
users need to be made aware of it -- probably by reading docs or readmes
which is what consistently doesn't happen.
-Kirk
On Fri, Jan 8, 2016 at 10:44 AM, Bruce Schuchardt <bs...@pivotal.io>
wrote:
> The default member-timeout is 5 seconds. For an unpredictable network or
> a system with GC pauses we might want to use a longer member-timeout in
> deployment. Network-partition-detection isn't involved in that though -
> it's just normal failure detection.
>
> Where network-partition-detection would cause harm is in a small
> deployment: say 2 servers & 1 locator. If the "lead" server is kicked out
> this would cause both the locator and other server to shut-down because the
> membership weight was 28 and 15 of that was lost. They would all restart
> after a default delay of 1 minute using the auto-reconnect feature, which
> is enabled by default.
>
>
>
> Le 1/8/2016 8:13 AM, Real Wes Williams a écrit :
>
>> What’s the level of concern here about members getting kicked out
>> prematurely depending on the newly proposed default settings? For
>> instance, if the default suspect notification is 3 seconds and they are
>> running in AWS or a mildly unpredictable network environment, a member
>> could be kicked out. What would be considered “safe” settings?
>>
>> On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
>>> wrote:
>>>
>>> Another thing that's been discussed for a long time is turning on
>>> network-partition-detection by default. It is a major problem for someone
>>> if a partition occurs and they are using persistence. The disk-stores on
>>> all but one of the partitions have to be deleted and revoked.
>>>
>>
>
Re: enabling network partition detection by default
Posted by Bruce Schuchardt <bs...@pivotal.io>.
The default member-timeout is 5 seconds. For an unpredictable network
or a system with GC pauses we might want to use a longer member-timeout
in deployment. Network-partition-detection isn't involved in that
though - it's just normal failure detection.
Where network-partition-detection would cause harm is in a small
deployment: say 2 servers & 1 locator. If the "lead" server is kicked
out this would cause both the locator and other server to shut-down
because the membership weight was 28 and 15 of that was lost. They
would all restart after a default delay of 1 minute using the
auto-reconnect feature, which is enabled by default.
Le 1/8/2016 8:13 AM, Real Wes Williams a écrit :
> What’s the level of concern here about members getting kicked out prematurely depending on the newly proposed default settings? For instance, if the default suspect notification is 3 seconds and they are running in AWS or a mildly unpredictable network environment, a member could be kicked out. What would be considered “safe” settings?
>
>> On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io> wrote:
>>
>> Another thing that's been discussed for a long time is turning on network-partition-detection by default. It is a major problem for someone if a partition occurs and they are using persistence. The disk-stores on all but one of the partitions have to be deleted and revoked.
Re: enabling network partition detection by default
Posted by Dan Smith <ds...@pivotal.io>.
> On the diskstore topic, if we know that the disk stores are wrong at
> startup, why don't we just fix them? Why do we require user intervention?
>
>
Let's be clear on what "wrong" means here. If you don't have network
partition detection turned on, that means that you can end up with multiple
copies of the same region that you are updating independently. In other
words, your copies are no longer consistent - you've made different edits
to these different copies.
The disk stores detect this condition on restart and report an error, to
protect you from losing data. The only "fix" we could do is to switch to an
eventual consistency model where there is way to merge and resolve
conflicts between different versions of a bucket or a region.
Supporting an eventual consistency model is something we could do, but
that's not the path we've gone down - we've been favoring consistency over
availability. If we really wanted to support eventual consistency we would
not have network partition detection, we would just allow the modifications
to continue even if the members are not connected and then support some way
of merging the data when they are reconnected.
-Dan
Re: enabling network partition detection by default
Posted by Swapnil Bawaskar <sb...@pivotal.io>.
I don't like the idea of having two properties file as it will be confusing
to both new and existing users.
Even if we decide to go with two files, I think we should strive to keep
things simpler for new users, say by have a "old-defaults.properties" file
which the existing users can use.
On Fri, Jan 8, 2016 at 8:34 AM, Michael Stolz <ms...@pivotal.io> wrote:
> I'm not sure I want to do wholesale defaults revision.
>
> Yes this is a release that cannot be done in a rolling fashion, but it is
> not (yet) a release that requires that pre-existing users to change their
> settings in general to keep their current configuration.
>
> I think it would be simple enough to supply a "recommended.properties" file
> that sets things the way we now like them without actually changing the
> hard-coded defaults.
>
> I view the network-partition-detection property as a special case because
> of its affect on diskstores which are automatically created if you use the
> recommended serialization mechanism (pdx).
>
> On the diskstore topic, if we know that the disk stores are wrong at
> startup, why don't we just fix them? Why do we require user intervention?
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Fri, Jan 8, 2016 at 11:13 AM, Real Wes Williams <TheRealWes@outlook.com
> >
> wrote:
>
> > What’s the level of concern here about members getting kicked out
> > prematurely depending on the newly proposed default settings? For
> > instance, if the default suspect notification is 3 seconds and they are
> > running in AWS or a mildly unpredictable network environment, a member
> > could be kicked out. What would be considered “safe” settings?
> >
> > > On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
> > wrote:
> > >
> > > Another thing that's been discussed for a long time is turning on
> > network-partition-detection by default. It is a major problem for
> someone
> > if a partition occurs and they are using persistence. The disk-stores on
> > all but one of the partitions have to be deleted and revoked.
> >
> >
>
Re: enabling network partition detection by default
Posted by Bruce Schuchardt <bs...@pivotal.io>.
Persistence engineers, please correct me if I'm wrong: The diskStores
record membership view information and use this to figure out which
store is most up-to-date. When there is a split-brain there's no way to
know which is more recent.
Le 1/8/2016 8:34 AM, Michael Stolz a écrit :
> I'm not sure I want to do wholesale defaults revision.
>
> Yes this is a release that cannot be done in a rolling fashion, but it is
> not (yet) a release that requires that pre-existing users to change their
> settings in general to keep their current configuration.
>
> I think it would be simple enough to supply a "recommended.properties" file
> that sets things the way we now like them without actually changing the
> hard-coded defaults.
>
> I view the network-partition-detection property as a special case because
> of its affect on diskstores which are automatically created if you use the
> recommended serialization mechanism (pdx).
>
> On the diskstore topic, if we know that the disk stores are wrong at
> startup, why don't we just fix them? Why do we require user intervention?
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Fri, Jan 8, 2016 at 11:13 AM, Real Wes Williams <Th...@outlook.com>
> wrote:
>
>> What’s the level of concern here about members getting kicked out
>> prematurely depending on the newly proposed default settings? For
>> instance, if the default suspect notification is 3 seconds and they are
>> running in AWS or a mildly unpredictable network environment, a member
>> could be kicked out. What would be considered “safe” settings?
>>
>>> On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
>> wrote:
>>> Another thing that's been discussed for a long time is turning on
>> network-partition-detection by default. It is a major problem for someone
>> if a partition occurs and they are using persistence. The disk-stores on
>> all but one of the partitions have to be deleted and revoked.
>>
>>
Re: enabling network partition detection by default
Posted by Michael Stolz <ms...@pivotal.io>.
I'm not sure I want to do wholesale defaults revision.
Yes this is a release that cannot be done in a rolling fashion, but it is
not (yet) a release that requires that pre-existing users to change their
settings in general to keep their current configuration.
I think it would be simple enough to supply a "recommended.properties" file
that sets things the way we now like them without actually changing the
hard-coded defaults.
I view the network-partition-detection property as a special case because
of its affect on diskstores which are automatically created if you use the
recommended serialization mechanism (pdx).
On the diskstore topic, if we know that the disk stores are wrong at
startup, why don't we just fix them? Why do we require user intervention?
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Fri, Jan 8, 2016 at 11:13 AM, Real Wes Williams <Th...@outlook.com>
wrote:
> What’s the level of concern here about members getting kicked out
> prematurely depending on the newly proposed default settings? For
> instance, if the default suspect notification is 3 seconds and they are
> running in AWS or a mildly unpredictable network environment, a member
> could be kicked out. What would be considered “safe” settings?
>
> > On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io>
> wrote:
> >
> > Another thing that's been discussed for a long time is turning on
> network-partition-detection by default. It is a major problem for someone
> if a partition occurs and they are using persistence. The disk-stores on
> all but one of the partitions have to be deleted and revoked.
>
>
Re: enabling network partition detection by default
Posted by Real Wes Williams <Th...@outlook.com>.
What’s the level of concern here about members getting kicked out prematurely depending on the newly proposed default settings? For instance, if the default suspect notification is 3 seconds and they are running in AWS or a mildly unpredictable network environment, a member could be kicked out. What would be considered “safe” settings?
> On Jan 7, 2016, at 4:18 PM, Bruce Schuchardt <bs...@pivotal.io> wrote:
>
> Another thing that's been discussed for a long time is turning on network-partition-detection by default. It is a major problem for someone if a partition occurs and they are using persistence. The disk-stores on all but one of the partitions have to be deleted and revoked.