You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Carlos Rolo <ro...@pythian.com> on 2015/05/11 11:17:50 UTC
Nodes failed to bootstrap, no nodetool info but system.peer populated.
Hi all,
I just wanted to know if this should be worth filling a bug or not
(Couldn't find any similar).
I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
because of hardware failure (virtualized environment).
The process was automated, so what was supposed to happen was:
- Node 4 joins
- wait until status is UN and then 2min more
- Node 5 joins
- wait until status is UN and then 2min more
- Node 6 joins
- wait until status is UN and then 2min more
What happened:
- Node 4 joins
- Wait...
- Node 5 joins
- VM fails while node is starting.
- VM 6 starts, no node with UN, waits 2min
- Node 6 joins
- VM fails while node is starting.
After this, nodetool reports 4 nodes all UN
While trying an application (Datastax Java Driver 2.1) the debug log
reports that it tries to connect to Node 5 and 6 and fails.
Checking system.peers table, I see both nodes there. So I tried "nodetool
removenode <ID>" with the IDs in the table.
It blows up with the following exception:
Exception in thread "main" java.lang.UnsupportedOperationException: Host ID
not found.
Then I decided to do the following:
DELETE from peers where ID in (ID1, ID2);
All good, cluster still happy and driver not complaining anymore.
Is this expected behavior?
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
--
--
Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.
Posted by Carlos Rolo <ro...@pythian.com>.
Thanks also!
I did it, JAVA-761 <https://datastax-oss.atlassian.net/browse/JAVA-761>
created!
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Mon, May 11, 2015 at 6:48 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> I hit this issue today with the c# driver. I still think the drivers should
> handle peers inconsistencies better and maybe even output warnings about
> them.
>
> I opened CSHARP-296, @rolo, it's probably a good idea to open a similar one
> for java.
> On May 11, 2015 11:24 AM, "Carlos Rolo" <ro...@pythian.com> wrote:
>
> > Thanks!
> >
> > Regards,
> >
> > Carlos Juzarte Rolo
> > Cassandra Consultant
> >
> > Pythian - Love your data
> >
> > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > linkedin.com/in/carlosjuzarterolo
> > <http://linkedin.com/in/carlosjuzarterolo>*
> > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > www.pythian.com
> >
> > On Mon, May 11, 2015 at 4:29 PM, Brandon Williams <dr...@gmail.com>
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/CASSANDRA-9180
> > >
> > > On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo <ro...@pythian.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I just wanted to know if this should be worth filling a bug or not
> > > > (Couldn't find any similar).
> > > >
> > > > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > > > because of hardware failure (virtualized environment).
> > > >
> > > > The process was automated, so what was supposed to happen was:
> > > >
> > > > - Node 4 joins
> > > > - wait until status is UN and then 2min more
> > > > - Node 5 joins
> > > > - wait until status is UN and then 2min more
> > > > - Node 6 joins
> > > > - wait until status is UN and then 2min more
> > > >
> > > > What happened:
> > > > - Node 4 joins
> > > > - Wait...
> > > > - Node 5 joins
> > > > - VM fails while node is starting.
> > > > - VM 6 starts, no node with UN, waits 2min
> > > > - Node 6 joins
> > > > - VM fails while node is starting.
> > > >
> > > > After this, nodetool reports 4 nodes all UN
> > > > While trying an application (Datastax Java Driver 2.1) the debug log
> > > > reports that it tries to connect to Node 5 and 6 and fails.
> > > >
> > > > Checking system.peers table, I see both nodes there. So I tried
> > "nodetool
> > > > removenode <ID>" with the IDs in the table.
> > > >
> > > > It blows up with the following exception:
> > > > Exception in thread "main" java.lang.UnsupportedOperationException:
> > Host
> > > ID
> > > > not found.
> > > >
> > > > Then I decided to do the following:
> > > > DELETE from peers where ID in (ID1, ID2);
> > > >
> > > > All good, cluster still happy and driver not complaining anymore.
> > > > Is this expected behavior?
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Carlos Juzarte Rolo
> > > > Cassandra Consultant
> > > >
> > > > Pythian - Love your data
> > > >
> > > > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > > > linkedin.com/in/carlosjuzarterolo
> > > > <http://linkedin.com/in/carlosjuzarterolo>*
> > > > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > > > www.pythian.com
> > > >
> > > > --
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > >
> > > >
> > >
> >
> > --
> >
> >
> > --
> >
> >
> >
> >
>
--
--
Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.
Posted by Sebastian Estevez <se...@datastax.com>.
I hit this issue today with the c# driver. I still think the drivers should
handle peers inconsistencies better and maybe even output warnings about
them.
I opened CSHARP-296, @rolo, it's probably a good idea to open a similar one
for java.
On May 11, 2015 11:24 AM, "Carlos Rolo" <ro...@pythian.com> wrote:
> Thanks!
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *
> linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, May 11, 2015 at 4:29 PM, Brandon Williams <dr...@gmail.com>
> wrote:
>
> > https://issues.apache.org/jira/browse/CASSANDRA-9180
> >
> > On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo <ro...@pythian.com> wrote:
> >
> > > Hi all,
> > >
> > > I just wanted to know if this should be worth filling a bug or not
> > > (Couldn't find any similar).
> > >
> > > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > > because of hardware failure (virtualized environment).
> > >
> > > The process was automated, so what was supposed to happen was:
> > >
> > > - Node 4 joins
> > > - wait until status is UN and then 2min more
> > > - Node 5 joins
> > > - wait until status is UN and then 2min more
> > > - Node 6 joins
> > > - wait until status is UN and then 2min more
> > >
> > > What happened:
> > > - Node 4 joins
> > > - Wait...
> > > - Node 5 joins
> > > - VM fails while node is starting.
> > > - VM 6 starts, no node with UN, waits 2min
> > > - Node 6 joins
> > > - VM fails while node is starting.
> > >
> > > After this, nodetool reports 4 nodes all UN
> > > While trying an application (Datastax Java Driver 2.1) the debug log
> > > reports that it tries to connect to Node 5 and 6 and fails.
> > >
> > > Checking system.peers table, I see both nodes there. So I tried
> "nodetool
> > > removenode <ID>" with the IDs in the table.
> > >
> > > It blows up with the following exception:
> > > Exception in thread "main" java.lang.UnsupportedOperationException:
> Host
> > ID
> > > not found.
> > >
> > > Then I decided to do the following:
> > > DELETE from peers where ID in (ID1, ID2);
> > >
> > > All good, cluster still happy and driver not complaining anymore.
> > > Is this expected behavior?
> > >
> > >
> > >
> > > Regards,
> > >
> > > Carlos Juzarte Rolo
> > > Cassandra Consultant
> > >
> > > Pythian - Love your data
> > >
> > > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > > linkedin.com/in/carlosjuzarterolo
> > > <http://linkedin.com/in/carlosjuzarterolo>*
> > > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > > www.pythian.com
> > >
> > > --
> > >
> > >
> > > --
> > >
> > >
> > >
> > >
> >
>
> --
>
>
> --
>
>
>
>
Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.
Posted by Carlos Rolo <ro...@pythian.com>.
Thanks!
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Mon, May 11, 2015 at 4:29 PM, Brandon Williams <dr...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-9180
>
> On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo <ro...@pythian.com> wrote:
>
> > Hi all,
> >
> > I just wanted to know if this should be worth filling a bug or not
> > (Couldn't find any similar).
> >
> > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > because of hardware failure (virtualized environment).
> >
> > The process was automated, so what was supposed to happen was:
> >
> > - Node 4 joins
> > - wait until status is UN and then 2min more
> > - Node 5 joins
> > - wait until status is UN and then 2min more
> > - Node 6 joins
> > - wait until status is UN and then 2min more
> >
> > What happened:
> > - Node 4 joins
> > - Wait...
> > - Node 5 joins
> > - VM fails while node is starting.
> > - VM 6 starts, no node with UN, waits 2min
> > - Node 6 joins
> > - VM fails while node is starting.
> >
> > After this, nodetool reports 4 nodes all UN
> > While trying an application (Datastax Java Driver 2.1) the debug log
> > reports that it tries to connect to Node 5 and 6 and fails.
> >
> > Checking system.peers table, I see both nodes there. So I tried "nodetool
> > removenode <ID>" with the IDs in the table.
> >
> > It blows up with the following exception:
> > Exception in thread "main" java.lang.UnsupportedOperationException: Host
> ID
> > not found.
> >
> > Then I decided to do the following:
> > DELETE from peers where ID in (ID1, ID2);
> >
> > All good, cluster still happy and driver not complaining anymore.
> > Is this expected behavior?
> >
> >
> >
> > Regards,
> >
> > Carlos Juzarte Rolo
> > Cassandra Consultant
> >
> > Pythian - Love your data
> >
> > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > linkedin.com/in/carlosjuzarterolo
> > <http://linkedin.com/in/carlosjuzarterolo>*
> > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > www.pythian.com
> >
> > --
> >
> >
> > --
> >
> >
> >
> >
>
--
--
Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.
Posted by Brandon Williams <dr...@gmail.com>.
https://issues.apache.org/jira/browse/CASSANDRA-9180
On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo <ro...@pythian.com> wrote:
> Hi all,
>
> I just wanted to know if this should be worth filling a bug or not
> (Couldn't find any similar).
>
> I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> because of hardware failure (virtualized environment).
>
> The process was automated, so what was supposed to happen was:
>
> - Node 4 joins
> - wait until status is UN and then 2min more
> - Node 5 joins
> - wait until status is UN and then 2min more
> - Node 6 joins
> - wait until status is UN and then 2min more
>
> What happened:
> - Node 4 joins
> - Wait...
> - Node 5 joins
> - VM fails while node is starting.
> - VM 6 starts, no node with UN, waits 2min
> - Node 6 joins
> - VM fails while node is starting.
>
> After this, nodetool reports 4 nodes all UN
> While trying an application (Datastax Java Driver 2.1) the debug log
> reports that it tries to connect to Node 5 and 6 and fails.
>
> Checking system.peers table, I see both nodes there. So I tried "nodetool
> removenode <ID>" with the IDs in the table.
>
> It blows up with the following exception:
> Exception in thread "main" java.lang.UnsupportedOperationException: Host ID
> not found.
>
> Then I decided to do the following:
> DELETE from peers where ID in (ID1, ID2);
>
> All good, cluster still happy and driver not complaining anymore.
> Is this expected behavior?
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *
> linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> --
>
>
> --
>
>
>
>