Cassandra DCOS | monitoring connection and user activity

["Caesar, Maik" <ma...@dxc.com>]

Hello All,
have someone experience with monitoring cassandra in DCOS?
If we increase the load to the Casandra in DCOS, the application get timeouts and loose the connection and I do not have any information about what happened.
Is there a way to get information about the amount of current connection and which queries are executed? Cassandra in DCOS has disabled the JMX interface and I think the noodetool do not provide such information.

Regards
Maik


DXC Technology Company -- This message is transmitted to you by or on behalf of DXC Technology Company or one of its affiliates. It is intended exclusively for the addressee. The substance of this message, along with any attachments, may contain proprietary, confidential or privileged information or information that is otherwise legally exempt from disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate any part of this message. If you have received this message in error, please destroy and delete all copies and notify the sender by return e-mail. Regardless of content, this e-mail shall not operate to bind DXC Technology Company or any of its affiliates to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose. --.

Re: Cassandra DCOS | monitoring connection and user activity

[Ben Bromhead <be...@instaclustr.com>]

To dig a little deeper.

It sounds like JMX is just enabled for loopback, if JMX was
disabled/unreachable entirely, nodetool wouldn't work at all.

You will likely need to customize your Cassandra config to allow remote JMX
connections (though this does increase your attack surface and risk from a
security perspective). Alternatively, whatever monitoring solution you land
on will likely need a local agent to connect to JMX via loopback.

To answer your specific monitoring questions, iirc JMX will expose the
number of active connections per node, however, it does not show what
queries are being executed.

Cassandra does support slow query logging (later versions), or you can use
query tracing (probabilistic and at the session level) to see what is
happening in real time. Though query tracing is fairly heavy from a
performance perspective.

I generally don't recommend thinking about tracking bad query issues in
Cassandra the same way you would with a relational DB. Cassandra does not
have long running queries that get stuck, hold resources, hold locks etc.

Most bad performance smells from bad queries (as opposed to just an
increase in load) will show up in the logs e.g. large batches, queries
timing out, slow queries (when slow query logger is enabled),
excessive/long GC pauses, etc

However, you mention you get timeouts when you increase the load on
Cassandra. If you had some monitoring in place to graph reads and writes
per second against timeouts you might discover you simply need to add more
nodes or look at trying to squeeze more performance out of what you have.

Refer to Anups previous thoughts on where to get started with getting
operational data out of Cassandra

On Wed, Oct 31, 2018 at 8:21 PM Anup Shirolkar <
anup.shirolkar@instaclustr.com> wrote:

> Hi,
>
> Looks like you need a monitoring for Cassandra but without using JMX.
> It is possible to use metric reporting libraries in Cassandra:
> https://wiki.apache.org/cassandra/Metrics#Reporting
>
> I do not have specific experience with using Cassandra on DCOS but,
> monitoring with libraries and tools should not be any different.
>
> There are various options available to establish good monitoring
> (Graphite, Prometheus, Grafana)
>
> Helpful links:
>
>
> https://blog.pythian.com/monitoring-apache-cassandra-metrics-graphite-grafana/
> https://github.com/instaclustr/cassandra-exporter
> https://prometheus.io/docs/instrumenting/exporters/
> https://grafana.com/dashboards/5408
>
> http://thelastpickle.com/blog/2017/04/05/cassandra-graphite-measurements-filtered.html
>
> Regards,
>
> Anup Shirolkar
>
>
>
>
> On Wed, 31 Oct 2018 at 18:41, Caesar, Maik <ma...@dxc.com> wrote:
>
>> Hello All,
>>
>> have someone experience with monitoring cassandra in DCOS?
>>
>> If we increase the load to the Casandra in DCOS, the application get
>> timeouts and loose the connection and I do not have any information about
>> what happened.
>>
>> Is there a way to get information about the amount of current connection
>> and which queries are executed? Cassandra in DCOS has disabled the JMX
>> interface and I think the noodetool do not provide such information.
>>
>>
>>
>> Regards
>>
>> Maik
>>
>>
>>
>> DXC Technology Company -- This message is transmitted to you by or on
>> behalf of DXC Technology Company or one of its affiliates. It is intended
>> exclusively for the addressee. The substance of this message, along with
>> any attachments, may contain proprietary, confidential or privileged
>> information or information that is otherwise legally exempt from
>> disclosure. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient of this message, you are
>> not authorized to read, print, retain, copy or disseminate any part of this
>> message. If you have received this message in error, please destroy and
>> delete all copies and notify the sender by return e-mail. Regardless of
>> content, this e-mail shall not operate to bind DXC Technology Company or
>> any of its affiliates to any order or other contract unless pursuant to
>> explicit written agreement or government initiative expressly permitting
>> the use of e-mail for such purpose. --.
>>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Cassandra DCOS | monitoring connection and user activity

[Anup Shirolkar <an...@instaclustr.com>]

Hi,

Looks like you need a monitoring for Cassandra but without using JMX.
It is possible to use metric reporting libraries in Cassandra:
https://wiki.apache.org/cassandra/Metrics#Reporting

I do not have specific experience with using Cassandra on DCOS but,
monitoring with libraries and tools should not be any different.

There are various options available to establish good monitoring (Graphite,
Prometheus, Grafana)

Helpful links:

https://blog.pythian.com/monitoring-apache-cassandra-metrics-graphite-grafana/
https://github.com/instaclustr/cassandra-exporter
https://prometheus.io/docs/instrumenting/exporters/
https://grafana.com/dashboards/5408
http://thelastpickle.com/blog/2017/04/05/cassandra-graphite-measurements-filtered.html

Regards,

Anup Shirolkar




On Wed, 31 Oct 2018 at 18:41, Caesar, Maik <ma...@dxc.com> wrote:

> Hello All,
>
> have someone experience with monitoring cassandra in DCOS?
>
> If we increase the load to the Casandra in DCOS, the application get
> timeouts and loose the connection and I do not have any information about
> what happened.
>
> Is there a way to get information about the amount of current connection
> and which queries are executed? Cassandra in DCOS has disabled the JMX
> interface and I think the noodetool do not provide such information.
>
>
>
> Regards
>
> Maik
>
>
>
> DXC Technology Company -- This message is transmitted to you by or on
> behalf of DXC Technology Company or one of its affiliates. It is intended
> exclusively for the addressee. The substance of this message, along with
> any attachments, may contain proprietary, confidential or privileged
> information or information that is otherwise legally exempt from
> disclosure. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient of this message, you are
> not authorized to read, print, retain, copy or disseminate any part of this
> message. If you have received this message in error, please destroy and
> delete all copies and notify the sender by return e-mail. Regardless of
> content, this e-mail shall not operate to bind DXC Technology Company or
> any of its affiliates to any order or other contract unless pursuant to
> explicit written agreement or government initiative expressly permitting
> the use of e-mail for such purpose. --.
>