You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David Montgomery <da...@gmail.com> on 2014/02/13 10:39:03 UTC
supervisord and cassandra
omg..wow...can cassandra run under supervisor?
this works from the command line as root
/var/apache-cassandra-<%=@version%>/bin/cassandra -f
this does not work. why?
[program:cassandra]
command = /var/apache-cassandra-<%=@version%>/bin/cassandra -f
#environment=JVM_EXTRA_OPTS="-Dcom.sun.management.jmxremote.port=8199"
autostart=true
autorestart=true
stopsignal=KILL
stopasgroup = true
killasgroup = true
stdout_logfile=/tmp/cassandra.log
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=10
stderr_logfile = /tmp/cassandra.err
stderr_logfile_maxbytes=10MB
stderr_logfile_backups=10
[group:cassandra_server]
programs=cassandra
cassandra: ERROR (abnormal termination)
/tmp/cassandra.err
Unrecognized VM option 'StringTableSize=1000003'
Could not create the Java virtual machine.
Unrecognized VM option 'StringTableSize=1000003'
Could not create the Java virtual machine.
Unrecognized VM option 'StringTableSize=1000003'
Could not create the Java virtual machine.
Unrecognized VM option 'StringTableSize=1000003'
Could not create the Java virtual machine.
java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Re: supervisord and cassandra
Posted by David Montgomery <da...@gmail.com>.
Thanks,
do you have a sample of the defaults? I just dont understand why from the
command line it works but not from supervisor
thanks
[include]
files = *.supervisor
[supervisord]
pidfile = /var/run/supervisord.pid
nodaemon=false
[supervisorctl]
serverurl=unix:///var/run/supervisord.sock ; use a unix:// URL for a unix
socket
[unix_http_server]
file = /var/run/supervisord.sock
[rpcinterface:supervisor]
supervisor.rpcinterface_factory =
supervisor.rpcinterface:make_main_rpcinterface
[include]
files = /etc/supervisor/conf.d/*.conf
On Thu, Feb 13, 2014 at 6:19 PM, Robert S <robert.stapenhurst@secondsync.com
> wrote:
> Hi David,
>
> We've had no problems running C* with supervisor, and your conf.d file is
> broadly similar to ours. I don't think those errors are anything to do with
> supervisor.
>
> We did however have issues starting C* with the recommended resource
> limits<http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installRecommendSettings.html>;
> if I remember correctly, the solution was to set supervisor's own defaults
> appropriately (/etc/defaults/supervisor). Also watch out for
> autostart/autorestart - if a node has been down for more than
> max_hint_window_ms<http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/dml/../configuration/configCassandra_yaml_r.html%23reference_ds_qfg_n1r_1k__max_hint_window_in_ms>you may not want supervisor to automatically restart the daemon.
>
> Good luck,
>
> Robert
>
>
> On 13 February 2014 09:39, David Montgomery <da...@gmail.com>wrote:
>
>> omg..wow...can cassandra run under supervisor?
>>
>> this works from the command line as root
>> /var/apache-cassandra-<%=@version%>/bin/cassandra -f
>>
>>
>> this does not work. why?
>>
>> [program:cassandra]
>> command = /var/apache-cassandra-<%=@version%>/bin/cassandra -f
>> #environment=JVM_EXTRA_OPTS="-Dcom.sun.management.jmxremote.port=8199"
>> autostart=true
>> autorestart=true
>>
>> stopsignal=KILL
>> stopasgroup = true
>> killasgroup = true
>>
>> stdout_logfile=/tmp/cassandra.log
>> stdout_logfile_maxbytes=100MB
>> stdout_logfile_backups=10
>>
>> stderr_logfile = /tmp/cassandra.err
>> stderr_logfile_maxbytes=10MB
>> stderr_logfile_backups=10
>>
>> [group:cassandra_server]
>> programs=cassandra
>>
>> cassandra: ERROR (abnormal termination)
>>
>> /tmp/cassandra.err
>>
>> Unrecognized VM option 'StringTableSize=1000003'
>> Could not create the Java virtual machine.
>> Unrecognized VM option 'StringTableSize=1000003'
>> Could not create the Java virtual machine.
>> Unrecognized VM option 'StringTableSize=1000003'
>> Could not create the Java virtual machine.
>> Unrecognized VM option 'StringTableSize=1000003'
>> Could not create the Java virtual machine.
>>
>> java -version
>> java version "1.7.0_25"
>> OpenJDK Runtime Environment (IcedTea 2.3.10)
>> (7u25-2.3.10-1ubuntu0.12.04.2)
>> OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
>>
>>
>>
>>
>>
>>
>
Re: supervisord and cassandra
Posted by Robert S <ro...@secondsync.com>.
Hi David,
We've had no problems running C* with supervisor, and your conf.d file is
broadly similar to ours. I don't think those errors are anything to do with
supervisor.
We did however have issues starting C* with the recommended resource
limits<http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installRecommendSettings.html>;
if I remember correctly, the solution was to set supervisor's own defaults
appropriately (/etc/defaults/supervisor). Also watch out for
autostart/autorestart - if a node has been down for more than
max_hint_window_ms<http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/dml/../configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__max_hint_window_in_ms>you
may not want supervisor to automatically restart the daemon.
Good luck,
Robert
On 13 February 2014 09:39, David Montgomery <da...@gmail.com>wrote:
> omg..wow...can cassandra run under supervisor?
>
> this works from the command line as root
> /var/apache-cassandra-<%=@version%>/bin/cassandra -f
>
>
> this does not work. why?
>
> [program:cassandra]
> command = /var/apache-cassandra-<%=@version%>/bin/cassandra -f
> #environment=JVM_EXTRA_OPTS="-Dcom.sun.management.jmxremote.port=8199"
> autostart=true
> autorestart=true
>
> stopsignal=KILL
> stopasgroup = true
> killasgroup = true
>
> stdout_logfile=/tmp/cassandra.log
> stdout_logfile_maxbytes=100MB
> stdout_logfile_backups=10
>
> stderr_logfile = /tmp/cassandra.err
> stderr_logfile_maxbytes=10MB
> stderr_logfile_backups=10
>
> [group:cassandra_server]
> programs=cassandra
>
> cassandra: ERROR (abnormal termination)
>
> /tmp/cassandra.err
>
> Unrecognized VM option 'StringTableSize=1000003'
> Could not create the Java virtual machine.
> Unrecognized VM option 'StringTableSize=1000003'
> Could not create the Java virtual machine.
> Unrecognized VM option 'StringTableSize=1000003'
> Could not create the Java virtual machine.
> Unrecognized VM option 'StringTableSize=1000003'
> Could not create the Java virtual machine.
>
> java -version
> java version "1.7.0_25"
> OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2)
> OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
>
>
>
>
>
>
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
So.. see the rest of my replies for a working configuration, but I
wanted to reply to your initial post.
What problem are you trying to solve, and why do you think using
supervisord to restart a failed c* node will help?
You really don't want a node to be bouncing up and down.. A dead or
dieing node should stay down, until you can troubleshoot *why* it is
dead or dieing and determine if it should be replaced by a new node, or
has a repairable issue that will allow you to rejoin it to the ring.
--
Kind regards,
Michael
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 08:32 PM, Michael Shuler wrote:
> On 02/14/2014 08:27 PM, Michael Shuler wrote:
>> On 02/14/2014 08:10 PM, Michael Shuler wrote:
>>> mshuler@debian:~$ sudo supervisorctl status
>>> cassandra_server:cassandra FATAL Exited too quickly
>>> (process log may have details)
>>
>> I imagine the problems all stem from the fact that the initializing
>> script, in my case, /opt/cassandra/bin/cassandra, is executed and it's
>> done starting c* (Exited too quickly), and the process that actually
>> needs to be supervised is the java process (thus the ignorance that it
>> is running, and the fact that killing it is not recognized).
>
> Yup.
> https://lists.supervisord.org/pipermail/supervisor-users/2012-December/001207.html
(Self reply again..)
with cassandra -f, which is not being backgrounded in the exec line of
the script, my conf works:
mshuler@debian:~$ sudo supervisorctl status
cassandra_server:cassandra RUNNING pid 2784, uptime 0:00:23
mshuler@debian:~$ pkill java
mshuler@debian:~$ ps axu|grep java
mshuler 2988 0.0 0.0 7828 876 pts/0 S+ 20:45 0:00 grep java
mshuler@debian:~$ ps axu|grep java
mshuler 2989 18.1 16.4 1056412 168492 ? Sl 20:45 0:04 java
-ea -javaagent:/opt/cassandra/bin/../lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities <...>
and in the supervisor log:
2014-02-14 20:42:23,067 INFO daemonizing the supervisord process
2014-02-14 20:42:23,067 INFO supervisord started with pid 2777
2014-02-14 20:42:24,072 INFO spawned: 'cassandra' with pid 2784
2014-02-14 20:42:39,302 INFO success: cassandra entered RUNNING state,
process has stayed up for > than 15 seconds (startsecs)
2014-02-14 20:45:57,131 INFO exited: cassandra (exit status 143; not
expected)
2014-02-14 20:45:58,134 INFO spawned: 'cassandra' with pid 2989
2014-02-14 20:46:13,241 INFO success: cassandra entered RUNNING state,
process has stayed up for > than 15 seconds (startsecs)
--
Michael
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 08:27 PM, Michael Shuler wrote:
> On 02/14/2014 08:10 PM, Michael Shuler wrote:
>> mshuler@debian:~$ sudo supervisorctl status
>> cassandra_server:cassandra FATAL Exited too quickly
>> (process log may have details)
>
> I imagine the problems all stem from the fact that the initializing
> script, in my case, /opt/cassandra/bin/cassandra, is executed and it's
> done starting c* (Exited too quickly), and the process that actually
> needs to be supervised is the java process (thus the ignorance that it
> is running, and the fact that killing it is not recognized).
Yup.
https://lists.supervisord.org/pipermail/supervisor-users/2012-December/001207.html
--
Michael
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 08:10 PM, Michael Shuler wrote:
> mshuler@debian:~$ sudo supervisorctl status
> cassandra_server:cassandra FATAL Exited too quickly (process log may have details)
I imagine the problems all stem from the fact that the initializing
script, in my case, /opt/cassandra/bin/cassandra, is executed and it's
done starting c* (Exited too quickly), and the process that actually
needs to be supervised is the java process (thus the ignorance that it
is running, and the fact that killing it is not recognized).
--
Michael
Re: supervisord and cassandra
Posted by David Montgomery <da...@gmail.com>.
I had to give up on supervisor. I installed the deb package rather than
from source. that worked though.
thanks
On Sat, Feb 15, 2014 at 10:10 AM, Michael Shuler <mi...@pbandjelly.org>wrote:
> On 02/14/2014 07:34 PM, Michael Shuler wrote:
>
>> On 02/14/2014 06:58 PM, David Montgomery wrote:
>>
>>> Hi,
>>>
>>> Using now oracle 7. commented out the line StringTableSize=1000003
>>> same issue. but nothing in the log file now.
>>>
>>> but I start from, the command line the works.
>>>
>>
>> What user are you running c* with, when running from the command line?
>> What user is running c* via supervisord?
>>
>
> So you peaked my interest and tried supervisord in a vm. I think you need
> to probably go hit up the supervisord community for some "how do I do this
> correctly" questions.
>
> Attached a console log and the conf I used. Here's what I did:
>
> - installed c* 2.0.5 with /var/{lib,log}/cassandra owned by my user, as
> usual
> - verified c* runs fine from the command line
> - killed c*
> - installed supervisor package and added the attached conf
> - stopped/started supervisord to pick up the new conf
> - c* is running fine and nodetool confirms
> - supervisorctl status shows ignorance of c* running (wrong config, I
> assume)
> - stopped supervisord, c* still running (not sure if this is normal..)
>
> I have never played with supervisord. It's interesting, but my guess is
> there is some additional magic needed by supervisor experts to help you
> with a properly behaving configuration.
>
> Good luck and do report back with a good config for the archives!
>
> --
> Kind regards,
> Michael
>
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 08:10 PM, Michael Shuler wrote:
> Attached a console log and the conf I used. Here's what I did:
>
> - installed c* 2.0.5 with /var/{lib,log}/cassandra owned by my user, as
> usual
> - verified c* runs fine from the command line
> - killed c*
> - installed supervisor package and added the attached conf
> - stopped/started supervisord to pick up the new conf
> - c* is running fine and nodetool confirms
> - supervisorctl status shows ignorance of c* running (wrong config, I
> assume)
I missed a couple steps in here:
- killed c* and supervisor never restarted it
- restarted supervisor service, which starts up c* fine
(not in my console paste)
> - stopped supervisord, c* still running (not sure if this is normal..)
(not in my console paste)
Anyway, let us know how this works out!
--
Michael
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 07:34 PM, Michael Shuler wrote:
> On 02/14/2014 06:58 PM, David Montgomery wrote:
>> Hi,
>>
>> Using now oracle 7. commented out the line StringTableSize=1000003
>> same issue. but nothing in the log file now.
>>
>> but I start from, the command line the works.
>
> What user are you running c* with, when running from the command line?
> What user is running c* via supervisord?
So you peaked my interest and tried supervisord in a vm. I think you
need to probably go hit up the supervisord community for some "how do I
do this correctly" questions.
Attached a console log and the conf I used. Here's what I did:
- installed c* 2.0.5 with /var/{lib,log}/cassandra owned by my user, as
usual
- verified c* runs fine from the command line
- killed c*
- installed supervisor package and added the attached conf
- stopped/started supervisord to pick up the new conf
- c* is running fine and nodetool confirms
- supervisorctl status shows ignorance of c* running (wrong config, I
assume)
- stopped supervisord, c* still running (not sure if this is normal..)
I have never played with supervisord. It's interesting, but my guess is
there is some additional magic needed by supervisor experts to help you
with a properly behaving configuration.
Good luck and do report back with a good config for the archives!
--
Kind regards,
Michael
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/14/2014 06:58 PM, David Montgomery wrote:
> Hi,
>
> Using now oracle 7. commented out the line StringTableSize=1000003
> same issue. but nothing in the log file now.
>
> but I start from, the command line the works.
What user are you running c* with, when running from the command line?
What user is running c* via supervisord?
--
Michael
Re: supervisord and cassandra
Posted by Robert Coli <rc...@eventbrite.com>.
FWIW, if you are using something like supervisor to ensure that cassandra
auto-starts if it crashes, you should be aware that there are good reasons
to not do that.
https://issues.apache.org/jira/browse/CASSANDRA-2356
Discusses some of the cases, but it basically sums to "starting a
distributed server has distributed side effects and should therefore be
done deliberately by a human".
=Rob
Re: supervisord and cassandra
Posted by David Montgomery <da...@gmail.com>.
Hi,
Using now oracle 7. commented out the line StringTableSize=1000003
same issue. but nothing in the log file now.
but I start from, the command line the works.
Thanks
On Fri, Feb 14, 2014 at 9:48 AM, Michael Shuler <mi...@pbandjelly.org>wrote:
> On 02/13/2014 07:03 PM, David Montgomery wrote:
>
>> I only added the -f flag after the first time it did not work. If I
>> dont use the -f flag.....
>>
>> cassandra_server:cassandra FATAL Exited too quickly (process
>> log may have details)
>>
>
> From your original message:
>
>
> > Unrecognized VM option 'StringTableSize=1000003'
> > Could not create the Java virtual machine.
>
> Comment out the -XX:StringTableSize=1000003 line in conf/cassandra-env.sh
> and see what happens.
>
>
> > java version "1.7.0_25"
> > OpenJDK Runtime Environment (IcedTea 2.3.10)
> (7u25-2.3.10-1ubuntu0.12.04.2)
>
> Use Oracle's JVM and see what happens.
>
> --
> Michael
>
Re: supervisord and cassandra
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 02/13/2014 07:03 PM, David Montgomery wrote:
> I only added the -f flag after the first time it did not work. If I
> dont use the -f flag.....
>
> cassandra_server:cassandra FATAL Exited too quickly (process
> log may have details)
From your original message:
> Unrecognized VM option 'StringTableSize=1000003'
> Could not create the Java virtual machine.
Comment out the -XX:StringTableSize=1000003 line in
conf/cassandra-env.sh and see what happens.
> java version "1.7.0_25"
> OpenJDK Runtime Environment (IcedTea 2.3.10)
(7u25-2.3.10-1ubuntu0.12.04.2)
Use Oracle's JVM and see what happens.
--
Michael
Re: supervisord and cassandra
Posted by Tristan Seligmann <mi...@mithrandi.net>.
On 14 February 2014 03:03, David Montgomery <da...@gmail.com> wrote:
> I only added the -f flag after the first time it did not work. If I dont
> use the -f flag.....
>
> cassandra_server:cassandra FATAL Exited too quickly (process log
> may have details)
Whoops, I got mixed up; the -f param to Cassandra means "don't fork",
not "fork", so this is indeed correct.
--
mithrandi, i Ainil en-Balandor, a faer Ambar
Re: supervisord and cassandra
Posted by David Montgomery <da...@gmail.com>.
Hi,
I only added the -f flag after the first time it did not work. If I dont
use the -f flag.....
cassandra_server:cassandra FATAL Exited too quickly (process log
may have details)
On Thu, Feb 13, 2014 at 11:25 PM, Tristan Seligmann <mithrandi@mithrandi.net
> wrote:
> On 13 February 2014 11:39, David Montgomery <da...@gmail.com>
> wrote:
> > [program:cassandra]
> > command = /var/apache-cassandra-<%=@version%>/bin/cassandra -f
>
> I'm not familiar with supervisor specifically, but most process
> supervisors want the supervised process not to fork; that is, remove
> the -f from the command here.
> --
> mithrandi, i Ainil en-Balandor, a faer Ambar
>
Re: supervisord and cassandra
Posted by Tristan Seligmann <mi...@mithrandi.net>.
On 13 February 2014 11:39, David Montgomery <da...@gmail.com> wrote:
> [program:cassandra]
> command = /var/apache-cassandra-<%=@version%>/bin/cassandra -f
I'm not familiar with supervisor specifically, but most process
supervisors want the supervised process not to fork; that is, remove
the -f from the command here.
--
mithrandi, i Ainil en-Balandor, a faer Ambar