You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Melvin Oosterveen <me...@hotmail.com> on 2023/01/16 13:52:42 UTC
Prometheus on Apache Tomcat multiple tomcat Instances
Hi all,
I'm currently working on Apache Tomcat stacks within my company and I would like to integrate Prometheus into our deployments. We have decided to go for the Advanced Configuration - Multiple Tomcat Instances on UNIX systems described in the RUNNING.txt.
Does anyone have some experience with the setup of the JMX Prometheus exporter using multiple Tomcat instances? My goal is to have a separate Java process (Prometheus) started when I start an Apache Tomcat instance.
So far I have downloaded the JMX exporter<https://github.com/prometheus/jmx_exporter> and added it to the created Tomcat Instance with a minimal Prometheus configuration.
```bash
[root@melvin prometheus]# cat config.yml
hostPort: localhost:9999
rules:
- pattern: ".*"
```
I have created a service file which allows me to start the Apache Tomcat instances separately from each other. Inside this configuration I tried to start the Prometheus Java process, but so far I have failed and the Java process won't expose itself on the configured port.
```bash
[root@melvin prometheus]# cat /etc/systemd/system/tomcat-10@.service
[Unit]
Description=Tomcat - instance %i
After=syslog.target network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
WorkingDirectory=/var/tomcat/%i
Environment="JAVA_HOME=/etc/alternatives/jre_openjdk"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml"
Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid"
Environment="CATALINA_BASE=/var/tomcat/%i/"
Environment="CATALINA_HOME=/opt/tomcat/10/"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
ExecStart=/opt/tomcat/10/bin/startup.sh
ExecStop=/opt/tomcat/10/bin/shutdown.sh
#RestartSec=10
#Restart=always
[Install]
WantedBy=multi-user.target
```
The instance then starts without errors and starts my sample.war application which I have downloaded from Apache.org. The initiated Java process is even shown.
```bash
tomcat 115910 1 0 Jan12 ? 00:02:24 /etc/alternatives/jre_openjdk/bin/java -Djava.util.logging.config.file=/var/tomcat/tomcat10//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath /opt/tomcat/10//bin/bootstrap.jar:/var/tomcat/tomcat10//bin/tomcat-juli.jar -Dcatalina.base=/var/tomcat/tomcat10/ -Dcatalina.home=/opt/tomcat/10/ -Djava.io.tmpdir=/var/tomcat/tomcat10//temp org.apache.catalina.startup.Bootstrap start
```
I wonder if anyone could help me out with the right setup of Prometheus on Apache Tomcat multiple tomcat Instances.
Thanks in advance,
Melvin Oosterveen
Re: Prometheus on Apache Tomcat multiple tomcat Instances
Posted by Melvin Oosterveen <me...@hotmail.com>.
Hi Peter,
Thank you for your help, I got it to work!
I have added the lines below to the tomcat service file now and it works. A separate JMX process is now started for each of the apache tomcat instances I start, this could cause some port conflicts though - looks like a new challange.
#Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10080 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom"
Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid"
Environment="CATALINA_BASE=/var/tomcat/%i/"
Environment="CATALINA_HOME=/opt/tomcat/10/"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC -javaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10082:/var/tomcat/tomcat10/prometheus/config.yml"
Another challenge is to get it to configure it to HTTPS and HTTP, but I think ill be able to manage that by adding --web.config.file to the configuration.
Kind regards,
Melvin
________________________________
From: logo@kreuser.name <lo...@kreuser.name>
Sent: Monday, January 16, 2023 6:05 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Prometheus on Apache Tomcat multiple tomcat Instances
Melvin,
> Am 16.01.2023 um 14:52 schrieb Melvin Oosterveen <me...@hotmail.com>:
>
>
> Hi all,
>
> I'm currently working on Apache Tomcat stacks within my company and I would like to integrate Prometheus into our deployments. We have decided to go for the Advanced Configuration - Multiple Tomcat Instances on UNIX systems described in the RUNNING.txt.
>
> Does anyone have some experience with the setup of the JMX Prometheus exporter using multiple Tomcat instances? My goal is to have a separate Java process (Prometheus) started when I start an Apache Tomcat instance.
>
in my case the prometheus java agent is tied 1:1 to a tomcat process. In my understanding, that would mean 1 prometheus exporter per tomcat (each on a different port of course).
> So far I have downloaded the JMX exporter<https://github.com/prometheus/jmx_exporter> and added it to the created Tomcat Instance with a minimal Prometheus configuration.
>
As per the github page jmxagent is the recommended way and not the standalone http server (which queries tomcat remotely on the jmx port).
Can you explain which one you are really trying to use?
> ```bash
>
> [root@melvin prometheus]# cat config.yml
>
> hostPort: localhost:9999
is tomcat listening on this port with jmx? You are trying to query jmx on this host on this port (your tomcat process). But did you start the prometheus exporter as a separate java process?
java -jar jmx_prometheus_httpserver-0.17.2.jar 12345 config.yaml
prometheus >> exporter:12345 >> tomcat:9999 (jmx)
>
> rules:
>
> - pattern: ".*"
>
> ```
>
> I have created a service file which allows me to start the Apache Tomcat instances separately from each other. Inside this configuration I tried to start the Prometheus Java process, but so far I have failed and the Java process won't expose itself on the configured port.
> ```bash
>
> [root@melvin prometheus]# cat /etc/systemd/system/tomcat-10@.service
>
> [Unit]
>
> Description=Tomcat - instance %i
>
> After=syslog.target network.target
>
>
> [Service]
>
> Type=forking
>
>
> User=tomcat
>
> Group=tomcat
>
>
> WorkingDirectory=/var/tomcat/%i
>
>
> Environment="JAVA_HOME=/etc/alternatives/jre_openjdk"
>
> Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml"
At this point you are using the jmxagent - inside the tomcat process. Then you MUST leave out the hostPort. Did you try to curl localhost:10080 ?
prometheus >> tomcat:10080 >> internal jmx lookup
If you really want to use the external exporter, you will have to configure jmx in JAVA_OPTS instead to run on port 9999. I'm not sure if the following is enough or too much:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
> Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid"
>
> Environment="CATALINA_BASE=/var/tomcat/%i/"
>
> Environment="CATALINA_HOME=/opt/tomcat/10/"
>
> Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
>
>
> ExecStart=/opt/tomcat/10/bin/startup.sh
>
> ExecStop=/opt/tomcat/10/bin/shutdown.sh
>
>
> #RestartSec=10
>
> #Restart=always
>
>
> [Install]
>
> WantedBy=multi-user.target
>
> ```
>
> The instance then starts without errors and starts my sample.war application which I have downloaded from Apache.org. The initiated Java process is even shown.
> ```bash
>
> tomcat 115910 1 0 Jan12 ? 00:02:24 /etc/alternatives/jre_openjdk/bin/java -Djava.util.logging.config.file=/var/tomcat/tomcat10//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath /opt/tomcat/10//bin/bootstrap.jar:/var/tomcat/tomcat10//bin/tomcat-juli.jar -Dcatalina.base=/var/tomcat/tomcat10/ -Dcatalina.home=/opt/tomcat/10/ -Djava.io.tmpdir=/var/tomcat/tomcat10//temp org.apache.catalina.startup.Bootstrap start
>
> ```
>
> I wonder if anyone could help me out with the right setup of Prometheus on Apache Tomcat multiple tomcat Instances.
>
I would go the javaagent route and get one tomcat fixed - then all others will work easily. To get authentication right, put a haproxy (or an Apache httpd if it already exists) in front of your exporter, so you can limit queries to your prometheus host.
> Thanks in advance,
>
> Melvin Oosterveen
Hope this helps.
Peter
Re: Prometheus on Apache Tomcat multiple tomcat Instances
Posted by lo...@kreuser.name.
Melvin,
> Am 16.01.2023 um 14:52 schrieb Melvin Oosterveen <me...@hotmail.com>:
>
>
> Hi all,
>
> I'm currently working on Apache Tomcat stacks within my company and I would like to integrate Prometheus into our deployments. We have decided to go for the Advanced Configuration - Multiple Tomcat Instances on UNIX systems described in the RUNNING.txt.
>
> Does anyone have some experience with the setup of the JMX Prometheus exporter using multiple Tomcat instances? My goal is to have a separate Java process (Prometheus) started when I start an Apache Tomcat instance.
>
in my case the prometheus java agent is tied 1:1 to a tomcat process. In my understanding, that would mean 1 prometheus exporter per tomcat (each on a different port of course).
> So far I have downloaded the JMX exporter<https://github.com/prometheus/jmx_exporter> and added it to the created Tomcat Instance with a minimal Prometheus configuration.
>
As per the github page jmxagent is the recommended way and not the standalone http server (which queries tomcat remotely on the jmx port).
Can you explain which one you are really trying to use?
> ```bash
>
> [root@melvin prometheus]# cat config.yml
>
> hostPort: localhost:9999
is tomcat listening on this port with jmx? You are trying to query jmx on this host on this port (your tomcat process). But did you start the prometheus exporter as a separate java process?
java -jar jmx_prometheus_httpserver-0.17.2.jar 12345 config.yaml
prometheus >> exporter:12345 >> tomcat:9999 (jmx)
>
> rules:
>
> - pattern: ".*"
>
> ```
>
> I have created a service file which allows me to start the Apache Tomcat instances separately from each other. Inside this configuration I tried to start the Prometheus Java process, but so far I have failed and the Java process won't expose itself on the configured port.
> ```bash
>
> [root@melvin prometheus]# cat /etc/systemd/system/tomcat-10@.service
>
> [Unit]
>
> Description=Tomcat - instance %i
>
> After=syslog.target network.target
>
>
> [Service]
>
> Type=forking
>
>
> User=tomcat
>
> Group=tomcat
>
>
> WorkingDirectory=/var/tomcat/%i
>
>
> Environment="JAVA_HOME=/etc/alternatives/jre_openjdk"
>
> Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml"
At this point you are using the jmxagent - inside the tomcat process. Then you MUST leave out the hostPort. Did you try to curl localhost:10080 ?
prometheus >> tomcat:10080 >> internal jmx lookup
If you really want to use the external exporter, you will have to configure jmx in JAVA_OPTS instead to run on port 9999. I'm not sure if the following is enough or too much:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
> Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid"
>
> Environment="CATALINA_BASE=/var/tomcat/%i/"
>
> Environment="CATALINA_HOME=/opt/tomcat/10/"
>
> Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
>
>
> ExecStart=/opt/tomcat/10/bin/startup.sh
>
> ExecStop=/opt/tomcat/10/bin/shutdown.sh
>
>
> #RestartSec=10
>
> #Restart=always
>
>
> [Install]
>
> WantedBy=multi-user.target
>
> ```
>
> The instance then starts without errors and starts my sample.war application which I have downloaded from Apache.org. The initiated Java process is even shown.
> ```bash
>
> tomcat 115910 1 0 Jan12 ? 00:02:24 /etc/alternatives/jre_openjdk/bin/java -Djava.util.logging.config.file=/var/tomcat/tomcat10//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.egd=file:///dev/urandom -Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath /opt/tomcat/10//bin/bootstrap.jar:/var/tomcat/tomcat10//bin/tomcat-juli.jar -Dcatalina.base=/var/tomcat/tomcat10/ -Dcatalina.home=/opt/tomcat/10/ -Djava.io.tmpdir=/var/tomcat/tomcat10//temp org.apache.catalina.startup.Bootstrap start
>
> ```
>
> I wonder if anyone could help me out with the right setup of Prometheus on Apache Tomcat multiple tomcat Instances.
>
I would go the javaagent route and get one tomcat fixed - then all others will work easily. To get authentication right, put a haproxy (or an Apache httpd if it already exists) in front of your exporter, so you can limit queries to your prometheus host.
> Thanks in advance,
>
> Melvin Oosterveen
Hope this helps.
Peter