You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stephen Portanova <sp...@gmail.com> on 2014/08/27 10:50:16 UTC
Can't Add AWS Node due to /mnt/cassandra/data directory
I already have a 3node m3.large DSE cluster, but I can't seem to add
another m3.large node. I'm using the
ubuntu-trusty-14.04-amd64-server-20140607.1
(ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and
the JNA, then I go into opscenter to add a node. Things look good for 3 or
4 green circles, until I either get this error: Start Errored: Timed out
waiting for Cassandra to start. or this error: Agent Connection Errored:
Timed out waiting for agent to connect.
I check the system.log and output.log, and they both say:
INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall
successful
ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory
/mnt/cassandra/data doesn't exist*
*ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has
no permission to create /mnt/cassandra/data directory*
INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE
shutting down...
ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199)
Exception in thread Thread[Thread-1,5,main]
java.lang.AssertionError
at
org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
My agent.log file says:
Node is still provisioning, not attempting to determine ip.
INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before
trying to determine IP over JMX again
INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning,
not attempting to determine ip.
INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before
trying to determine IP over JMX again
INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning,
not attempting to determine ip.
INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before
trying to determine IP over JMX again
INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning,
not attempting to determine ip.
I feel like I'm missing something easy with the mount, so if you could
point me in the right direction, I would really appreciate it!
--
Stephen Portanova
(480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Posted by Stephen Portanova <sp...@gmail.com>.
Awesome, thanks Ben!
On Wed, Aug 27, 2014 at 9:14 PM, Ben Bromhead <be...@instaclustr.com> wrote:
> Make sure you have also setup the ephemeral drives as a raid device (use
> mdadm) and mounted it under /mnt/cassandra otherwise your data dir is the
> os partition which is usually very small.
>
> Ben Bromhead
> Instaclustr | www.instaclustr.com | @instaclustr
> <http://twitter.com/instaclustr> | +61 415 936 359
>
> On 27 Aug 2014, at 8:21 pm, Stephen Portanova <sp...@gmail.com> wrote:
>
> Worked great! Thanks Mark!
>
>
> On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy <ma...@gmail.com>
> wrote:
>
>> Hi stephen,
>>
>> I have never added a node via OpsCenter, so this may be a short coming of
>> that process. However in non OpsCenter installs you would have to create
>> the data directories first:
>>
>> sudo mkdir -p /mnt/cassandra/commitlog
>> sudo mkdir -p /mnt/cassandra/data
>> sudo mkdir -p /mnt/cassandra/saved_caches
>>
>> And then give the cassandra user ownership of those directories:
>>
>> sudo chown -R "cassandra:cassandra" /mnt/cassandra
>>
>> Once this is done Cassandra will have the correct directories and
>> permission to start up.
>>
>>
>> Mark
>>
>>
>> On 27 August 2014 09:50, Stephen Portanova <sp...@gmail.com> wrote:
>>
>>> I already have a 3node m3.large DSE cluster, but I can't seem to add
>>> another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1
>>> (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7
>>> and the JNA, then I go into opscenter to add a node. Things look good for 3
>>> or 4 green circles, until I either get this error: Start Errored: Timed
>>> out waiting for Cassandra to start. or this error: Agent Connection
>>> Errored: Timed out waiting for agent to connect.
>>>
>>> I check the system.log and output.log, and they both say:
>>> INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA
>>> mlockall successful
>>> ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory
>>> /mnt/cassandra/data doesn't exist*
>>> *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239)
>>> Has no permission to create /mnt/cassandra/data directory*
>>> INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE
>>> shutting down...
>>> ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199)
>>> Exception in thread Thread[Thread-1,5,main]
>>> java.lang.AssertionError
>>> at
>>> org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
>>> at
>>> com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
>>> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
>>> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
>>>
>>> My agent.log file says:
>>>
>>> Node is still provisioning, not attempting to determine ip.
>>>
>>> INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before
>>> trying to determine IP over JMX again
>>>
>>> INFO [Initialization] 2014-08-27 08:41:17,849 Node is still
>>> provisioning, not attempting to determine ip.
>>>
>>> INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before
>>> trying to determine IP over JMX again
>>>
>>> INFO [Initialization] 2014-08-27 08:41:37,849 Node is still
>>> provisioning, not attempting to determine ip.
>>>
>>> INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before
>>> trying to determine IP over JMX again
>>>
>>> INFO [Initialization] 2014-08-27 08:41:57,850 Node is still
>>> provisioning, not attempting to determine ip.
>>>
>>>
>>> I feel like I'm missing something easy with the mount, so if you could
>>> point me in the right direction, I would really appreciate it!
>>>
>>> --
>>> Stephen Portanova
>>> (480) 495-2634
>>>
>>
>>
>
>
> --
> Stephen Portanova
> (480) 495-2634
>
>
>
--
Stephen Portanova
(480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Posted by Ben Bromhead <be...@instaclustr.com>.
Make sure you have also setup the ephemeral drives as a raid device (use mdadm) and mounted it under /mnt/cassandra otherwise your data dir is the os partition which is usually very small.
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
On 27 Aug 2014, at 8:21 pm, Stephen Portanova <sp...@gmail.com> wrote:
> Worked great! Thanks Mark!
>
>
> On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy <ma...@gmail.com> wrote:
> Hi stephen,
>
> I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first:
>
> sudo mkdir -p /mnt/cassandra/commitlog
> sudo mkdir -p /mnt/cassandra/data
> sudo mkdir -p /mnt/cassandra/saved_caches
>
> And then give the cassandra user ownership of those directories:
>
> sudo chown -R "cassandra:cassandra" /mnt/cassandra
>
> Once this is done Cassandra will have the correct directories and permission to start up.
>
>
> Mark
>
>
> On 27 August 2014 09:50, Stephen Portanova <sp...@gmail.com> wrote:
> I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect.
>
> I check the system.log and output.log, and they both say:
> INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful
> ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) Directory /mnt/cassandra/data doesn't exist
> ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory
> INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down...
> ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
> at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
>
> My agent.log file says:
> Node is still provisioning, not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip.
>
>
>
> I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it!
>
> --
> Stephen Portanova
> (480) 495-2634
>
>
>
>
> --
> Stephen Portanova
> (480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Posted by Stephen Portanova <sp...@gmail.com>.
Worked great! Thanks Mark!
On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy <ma...@gmail.com> wrote:
> Hi stephen,
>
> I have never added a node via OpsCenter, so this may be a short coming of
> that process. However in non OpsCenter installs you would have to create
> the data directories first:
>
> sudo mkdir -p /mnt/cassandra/commitlog
> sudo mkdir -p /mnt/cassandra/data
> sudo mkdir -p /mnt/cassandra/saved_caches
>
> And then give the cassandra user ownership of those directories:
>
> sudo chown -R "cassandra:cassandra" /mnt/cassandra
>
> Once this is done Cassandra will have the correct directories and
> permission to start up.
>
>
> Mark
>
>
> On 27 August 2014 09:50, Stephen Portanova <sp...@gmail.com> wrote:
>
>> I already have a 3node m3.large DSE cluster, but I can't seem to add
>> another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1
>> (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7
>> and the JNA, then I go into opscenter to add a node. Things look good for 3
>> or 4 green circles, until I either get this error: Start Errored: Timed
>> out waiting for Cassandra to start. or this error: Agent Connection
>> Errored: Timed out waiting for agent to connect.
>>
>> I check the system.log and output.log, and they both say:
>> INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall
>> successful
>> ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory
>> /mnt/cassandra/data doesn't exist*
>> *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has
>> no permission to create /mnt/cassandra/data directory*
>> INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE
>> shutting down...
>> ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199)
>> Exception in thread Thread[Thread-1,5,main]
>> java.lang.AssertionError
>> at
>> org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
>> at
>> com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
>> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
>> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
>>
>> My agent.log file says:
>>
>> Node is still provisioning, not attempting to determine ip.
>>
>> INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before
>> trying to determine IP over JMX again
>>
>> INFO [Initialization] 2014-08-27 08:41:17,849 Node is still
>> provisioning, not attempting to determine ip.
>>
>> INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before
>> trying to determine IP over JMX again
>>
>> INFO [Initialization] 2014-08-27 08:41:37,849 Node is still
>> provisioning, not attempting to determine ip.
>>
>> INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before
>> trying to determine IP over JMX again
>>
>> INFO [Initialization] 2014-08-27 08:41:57,850 Node is still
>> provisioning, not attempting to determine ip.
>>
>>
>> I feel like I'm missing something easy with the mount, so if you could
>> point me in the right direction, I would really appreciate it!
>>
>> --
>> Stephen Portanova
>> (480) 495-2634
>>
>
>
--
Stephen Portanova
(480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Posted by Mark Reddy <ma...@gmail.com>.
Hi stephen,
I have never added a node via OpsCenter, so this may be a short coming of
that process. However in non OpsCenter installs you would have to create
the data directories first:
sudo mkdir -p /mnt/cassandra/commitlog
sudo mkdir -p /mnt/cassandra/data
sudo mkdir -p /mnt/cassandra/saved_caches
And then give the cassandra user ownership of those directories:
sudo chown -R "cassandra:cassandra" /mnt/cassandra
Once this is done Cassandra will have the correct directories and
permission to start up.
Mark
On 27 August 2014 09:50, Stephen Portanova <sp...@gmail.com> wrote:
> I already have a 3node m3.large DSE cluster, but I can't seem to add
> another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1
> (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7
> and the JNA, then I go into opscenter to add a node. Things look good for 3
> or 4 green circles, until I either get this error: Start Errored: Timed
> out waiting for Cassandra to start. or this error: Agent Connection
> Errored: Timed out waiting for agent to connect.
>
> I check the system.log and output.log, and they both say:
> INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall
> successful
> ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory
> /mnt/cassandra/data doesn't exist*
> *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has
> no permission to create /mnt/cassandra/data directory*
> INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE
> shutting down...
> ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199)
> Exception in thread Thread[Thread-1,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
> at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
>
> My agent.log file says:
>
> Node is still provisioning, not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before
> trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning,
> not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before
> trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning,
> not attempting to determine ip.
>
> INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before
> trying to determine IP over JMX again
>
> INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning,
> not attempting to determine ip.
>
>
> I feel like I'm missing something easy with the mount, so if you could
> point me in the right direction, I would really appreciate it!
>
> --
> Stephen Portanova
> (480) 495-2634
>