You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zipporah <fl...@gmail.com> on 2020/04/07 21:52:23 UTC
prometheus jmx scrape failed
Hi,
I am new to Ignite and have a question regarding prometheus jmx exporter.
So I'm integrating Apache ignite with Apache Spark to accelerate the
performance of Spark applications. As part of monitoring, I'm using
prometheus jmx exporter to export metrics to prometheus.
My prometheus and metrics configurations are like this:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/conf/prometheus.yaml
This is working fine with ignite 2.7.x. Starting from ignite 2.8, in spark
application log I'm seeing following errors:
2020-04-06T16:26:01.595 [pool-1-thread-2hread] ERROR
prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector - JMX scrape failed:
java.lang.IllegalArgumentException: Not an Attribute:
javax.management.openmbean.TabularDataSupport(tabularType=javax.management.openmbean.TabularType(name=*org.apache.ignite.spi.systemview.view.ClientConnectionView*,rowType=javax.management.openmbean.CompositeType(name=org.apache.ignite.spi.systemview.view.ClientConnectionView,items=((itemName=connectionId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=localAddress,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=remoteAddress,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=systemViewRowId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=type,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=user,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=version,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)))),indexNames=(systemViewRowId)),contents={})
at javax.management.AttributeList.adding(AttributeList.java:328)
at javax.management.AttributeList.adding(AttributeList.java:335)
at javax.management.AttributeList.asList(AttributeList.java:165)
at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:156)
at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117)
at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:468)
at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at
io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at
io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at
sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I have no clue why this is happening. Is there anything wrong with the mbean
exposed by ignite?
Thanks,
Zippo
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: prometheus jmx scrape failed
Posted by zipporah <fl...@gmail.com>.
Thanks Nikolay. This explains why I'm seeing this error. I'll try your
suggestions.
Zipporah
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: prometheus jmx scrape failed
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Zipporah.
AFAIK Zabbix doesn’t support scraping of the tabular data JMX beans.
We add this kind of beans in 2.8.
It’s a system view beans [1]
By default, system views exported in the form of JMX beans and SQL views.
You can:
a. Disable system view JMX exporter via configuration IgniteConfiguration#setSystemViewExporterSpi
You may want to keep SQL exporter to be able to observe system views via an SQL interface.
This can be done like that
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setSystemViewExporterSpi(new SqlViewExporterSpi());
b. You can filter system views bean in Zabbix exporter [2]
[1] https://apacheignite.readme.io/docs/system-views
[2] https://www.zabbix.com/documentation/current/manual/discovery/low_level_discovery/jmx
> 8 апр. 2020 г., в 00:52, zipporah <fl...@gmail.com> написал(а):
>
> Hi,
>
> I am new to Ignite and have a question regarding prometheus jmx exporter.
>
> So I'm integrating Apache ignite with Apache Spark to accelerate the
> performance of Spark applications. As part of monitoring, I'm using
> prometheus jmx exporter to export metrics to prometheus.
>
> My prometheus and metrics configurations are like this:
> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/conf/prometheus.yaml
>
> This is working fine with ignite 2.7.x. Starting from ignite 2.8, in spark
> application log I'm seeing following errors:
>
> 2020-04-06T16:26:01.595 [pool-1-thread-2hread] ERROR
> prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector - JMX scrape failed:
> java.lang.IllegalArgumentException: Not an Attribute:
> javax.management.openmbean.TabularDataSupport(tabularType=javax.management.openmbean.TabularType(name=*org.apache.ignite.spi.systemview.view.ClientConnectionView*,rowType=javax.management.openmbean.CompositeType(name=org.apache.ignite.spi.systemview.view.ClientConnectionView,items=((itemName=connectionId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=localAddress,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=remoteAddress,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=systemViewRowId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=type,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=user,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=version,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)))),indexNames=(systemViewRowId)),contents={})
> at javax.management.AttributeList.adding(AttributeList.java:328)
> at javax.management.AttributeList.adding(AttributeList.java:335)
> at javax.management.AttributeList.asList(AttributeList.java:165)
> at
> io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:156)
> at
> io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117)
> at
> io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:468)
> at
> io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at
> io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
> at
> io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
> at
> io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
> at
> io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)
> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
> at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
> at
> sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
> at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> I have no clue why this is happening. Is there anything wrong with the mbean
> exposed by ignite?
>
> Thanks,
> Zippo
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/