Re: Cluster state ranges are all null after reboot

[Greg Pendlebury <gr...@gmail.com>]

Thanks again for the info. Hopefully we find some more clues if it
continues to occur. The ops team are looking at alternative deployment
methods as well, so we might end up avoiding the issue altogether.

Ta,
Greg


On 28 February 2014 02:42, Shalin Shekhar Mangar <sh...@gmail.com>wrote:

> I think it is just a side-effect of the current implementation that
> the ranges are assigned linearly. You can also verify this by choosing
> a document from each shard and running it's uniqueKey against the
> CompositeIdRouter's sliceHash method and verifying that it is included
> in the range.
>
> I couldn't reproduce this but I didn't try too hard either. If you are
> able to isolate a reproducible example then please do report back.
> I'll spend some time to review the related code again to see if I can
> spot the problem.
>
> On Thu, Feb 27, 2014 at 2:19 AM, Greg Pendlebury
> <gr...@gmail.com> wrote:
> > Thanks Shalin, that code might be helpful... do you know if there is a
> > reliable way to line up the ranges with the shard numbers? When the
> problem
> > occurred we had 80 million documents already in the index, and could not
> > issue even a basic 'deleteById' call. I'm tempted to assume they are just
> > assigned linearly since our Test and Prod clusters both look to work that
> > way now, but I can't be sure whether that is by design or just
> happenstance
> > of boot order.
> >
> > And no, unfortunately we have not been able to reproduce this issue
> > consistently despite trying a number of different things such as
> graceless
> > stop/start and screwing with the underlying WAR file (which is what we
> > thought puppet might be doing). The problem has occurred twice since, but
> > always in our Test environment. The fact that Test has only a single
> > replica per shard is the most likely culprit for me, but as mentioned,
> even
> > gracelessly killing the last replica in the cluster seems to leave the
> > range set correctly in clusterstate when we test it in isolation.
> >
> > In production (45 JVMs, 15 shards with 3 replicas each) we've never seen
> > the problem, despite a similar number of rollouts for version changes
> etc.
> >
> > Ta,
> > Greg
> >
> >
> >
> >
> > On 26 February 2014 23:46, Shalin Shekhar Mangar <shalinmangar@gmail.com
> >wrote:
> >
> >> If you have 15 shards and assuming that you've never used shard
> >> splitting, you can calculate the shard ranges by using new
> >> CompositeIdRouter().partitionRange(15, new
> >> CompositeIdRouter().fullRange())
> >>
> >> This gives me:
> >> [80000000-9110ffff, 91110000-a221ffff, a2220000-b332ffff,
> >> b3330000-c443ffff, c4440000-d554ffff, d5550000-e665ffff,
> >> e6660000-f776ffff, f7770000-887ffff, 8880000-1998ffff,
> >> 19990000-2aa9ffff, 2aaa0000-3bbaffff, 3bbb0000-4ccbffff,
> >> 4ccc0000-5ddcffff, 5ddd0000-6eedffff, 6eee0000-7fffffff]
> >>
> >> Have you done any more investigation into why this happened? Anything
> >> strange in the logs? Are you able to reproduce this in a test
> >> environment?
> >>
> >> On Wed, Feb 19, 2014 at 5:16 AM, Greg Pendlebury
> >> <gr...@gmail.com> wrote:
> >> > We've got a 15 shard cluster spread across 3 hosts. This morning our
> >> puppet
> >> > software rebooted them all and afterwards the 'range' for each shard
> has
> >> > become null in zookeeper. Is there any way to restore this value
> short of
> >> > rebuilding a fresh index?
> >> >
> >> > I've read various questions from people with a similar problem,
> although
> >> in
> >> > those cases it is usually a single shard that has become null allowing
> >> them
> >> > to infer what the value should be and manually fix it in ZK. In this
> >> case I
> >> > have no idea what the ranges should be. This is our test cluster, and
> >> > checking production I can see that the ranges don't appear to be
> >> > predictable based on the shard number.
> >> >
> >> > I'm also not certain why it even occurred. Our test cluster only has a
> >> > single replica per shard, so when a JVM is rebooted the cluster is
> >> > unavailable... would that cause this? Production has 3 replicas so we
> can
> >> > do rolling reboots.
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>