You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gil Ganz <gi...@gmail.com> on 2022/02/06 08:51:54 UTC
Running enablefullquerylog crashes cassandra
Hey
I'm trying to enable full query log on cassandra 4.01 node and it's causing
cassandra to shutdown
nodetool enablefullquerylog --path /mnt/fql_data
Cassandra has shutdown.
error: null
-- StackTrace --
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source)
at
org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836)
at
org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)
/mnt/fql_data is owned by cassandra user
Doesn't matter if directory is empty or not
contents of cassandra.yaml
full_query_logging_options:
log_dir: /mnt/fql_data
roll_cycle: HOURLY
block: true
max_queue_weight: 268435456
max_log_size: 34359738368
# archive command is "/path/to/script.sh %path" where %path is replaced
with the file being rolled:
# archive_command:
# max_archive_retries: 10
No errors in the log, last couple of lines are
INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334
BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll
cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log
size:34359738368 Archive command:
INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335
BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested
I noticed there is a similiar bug
https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried
setting disk_failure_policy to ignore, same thing.
Has Anyone encountered something similar?
gil
Re: Running enablefullquerylog crashes cassandra
Posted by Erick Ramirez <er...@datastax.com>.
Are there really no entries after those INFO messages? That indicates that
a person/script/daemon/tool/process killed Cassandra. Perhaps check the OS
logs to see if oom-killer kicked in to see if the C* process was
terminated. Cheers!
>
Re: Running enablefullquerylog crashes cassandra
Posted by Gil Ganz <gi...@gmail.com>.
Nothing in the cassandra logs (system/debug) last lines are
INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334
BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll
cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log
size:34359738368 Archive command:
INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335
BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested
On Sun, Feb 6, 2022 at 6:43 PM Jeff Jirsa <jj...@gmail.com> wrote:
> That looks like a nodetool stack - can you check the Cassandra log for
> corresponding error?
>
> On Feb 6, 2022, at 12:52 AM, Gil Ganz <gi...@gmail.com> wrote:
>
>
> Hey
> I'm trying to enable full query log on cassandra 4.01 node and it's
> causing cassandra to shutdown
>
> nodetool enablefullquerylog --path /mnt/fql_data
>
> Cassandra has shutdown.
> error: null
> -- StackTrace --
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
> at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
> at
> javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
> at
> javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
> at
> javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
> at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source)
> at
> org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836)
> at
> org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62)
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
> at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)
>
> /mnt/fql_data is owned by cassandra user
> Doesn't matter if directory is empty or not
>
> contents of cassandra.yaml
>
> full_query_logging_options:
> log_dir: /mnt/fql_data
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456
> max_log_size: 34359738368
> # archive command is "/path/to/script.sh %path" where %path is
> replaced with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
>
> No errors in the log, last couple of lines are
>
> INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334
> BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll
> cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log
> size:34359738368 Archive command:
> INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335
> BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested
>
> I noticed there is a similiar bug
> https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried
> setting disk_failure_policy to ignore, same thing.
> Has Anyone encountered something similar?
>
>
>
>
>
> gil
>
>
Re: Running enablefullquerylog crashes cassandra
Posted by Jeff Jirsa <jj...@gmail.com>.
That looks like a nodetool stack - can you check the Cassandra log for corresponding error?
> On Feb 6, 2022, at 12:52 AM, Gil Ganz <gi...@gmail.com> wrote:
>
>
> Hey
> I'm trying to enable full query log on cassandra 4.01 node and it's causing cassandra to shutdown
>
> nodetool enablefullquerylog --path /mnt/fql_data
>
> Cassandra has shutdown.
> error: null
> -- StackTrace --
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
> at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
> at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
> at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source)
> at org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836)
> at org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
> at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)
>
> /mnt/fql_data is owned by cassandra user
> Doesn't matter if directory is empty or not
>
> contents of cassandra.yaml
>
> full_query_logging_options:
> log_dir: /mnt/fql_data
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456
> max_log_size: 34359738368
> # archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
>
> No errors in the log, last couple of lines are
>
> INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log size:34359738368 Archive command:
> INFO [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested
>
> I noticed there is a similiar bug https://issues.apache.org/jira/browse/CASSANDRA-17136 but I also tried setting disk_failure_policy to ignore, same thing.
> Has Anyone encountered something similar?
>
>
>
>
>
> gil