You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID> on 2023/11/29 08:50:08 UTC
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
I am trying to add Tomcat-JMX Integration to the Datadog Agent, in order to achieve Remote Monitoring .
I am following the docs https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations
[https://datadog-docs.imgix.net/images/dd-docs-meta-image.png]<https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations>
Autodiscovery with JMX<https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations>
Datadog, the leading service for cloud-scale monitoring.
docs.datadoghq.com
This is the Environment I am using in my case.
I am trying to integrate JMX in one my Deployment, in AKS environment and I am using Helm to upgrade my environment.
as suggested in the documentation., I am adding the annotations in my deployment file in helm structure.
My agent is JMX enabled, and i can see in deployment logs.
I have configured a service for this deployment and its reflecting.
Issue I am facing :
I am trying to add my annotations, in the deployment file, after deployment, it should be able make the remote connection , but its showing these logs in the agent.
while i check the Java process., even its not reflecting this JMX process , it seems JMX process is not at all created, in my case.,
error i am facing after deployment"
========
JMXFetch
========
Information
==================
runtime_version : 11.0.18
version : 0.47.8
Initialized checks
==================
tomcat
instance_name : tomcat-10.75.12.143-1099
message : Unable to instantiate or initialize instance 10.75.12.143:1099. Is the target JMX Server or JVM running? Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.143; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
metric_count : 0
service_check_count : 0
status : ERROR
instance_name : tomcat-10.75.12.143-1099
message : Unable to instantiate or initialize instance 10.75.12.143:1099. Is the target JMX Server or JVM running? Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.143; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
metric_count : 0
service_check_count : 0
status : ERROR
Failed checks
=============
no checks
These are the annotations i have tried to implement JMX in my case:
ad.datadoghq.com/{{ .Values.configmap.app.name }}-container.checks: >-
{
"tomcat": {
"init_config": {
"is_jmx": true,
"collect_default_metrics": true
},
"instances": [{
"host": "%%host%%",
"port": "1099"
}]
}
}
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: JAVA_OPTS
value: >-
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=$(POD_IP)
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.host=$(POD_IP)
Even i tried by altering some of the unwanted annotations, I am not getting the expected result., ie., remote JMX connection.,
I would requet your help in this.,
Thanks in advance!
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Tuesday, November 28, 2023 19:41
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: [EXT] Re: Datadog _ JMX Integration facing connection issues.
External Email: Use caution with links and attachments.
Sai Vamsi,
On 11/28/23 04:29, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hello team,
> I am trying to add Tomcat-JMX Integration to the Datadog Agent, in order
> to achieve Remote Monitoring .
> I am following the docs
> https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!flrd6o4JKEdfTeHqFXZnxrdwVlDfgIQWa0LA1ItCAwqquaEQHF1TQrWhTdEdCz8b0U5R2hKiyy7adAu09KNW3__LBlrGJg7Il1X4ugDoVeSJPw$ <https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!flrd6o4JKEdfTeHqFXZnxrdwVlDfgIQWa0LA1ItCAwqquaEQHF1TQrWhTdEdCz8b0U5R2hKiyy7adAu09KNW3__LBlrGJg7Il1X4ugDoVeSJPw$ >
>
> *_This is the Environment I am using in my case._*
> I am trying to integrate JMX in one my Deployment, in AKS environment
> and I am using Helm to upgrade my environment.
> as suggested in the documentation., I am adding the annotations in my
> deployment file in helm structure.
> My agent is JMX enabled, and i can see in deployment logs.
> I have configured a service for this deployment and its reflecting.
>
>
> *_Issue I am facing :_*
> I am trying to add my annotations, in the deployment file, after
> deployment, it should be able make the remote connection , but its
> showing these logs in the agent.
> while i check the Java process., even its not reflecting this JMX
> process , it seems JMX process is not at all created, in my case.,
> *
> *
> *_Following are the annotations i am using in my case
>
> _*
> Even i tried by altering some of the unwanted annotations, I am not
> getting the expected result., ie., remote JMX connection.,
I'm sorry, but this list does not allow attached or embedded images. Can
you copy/paste your logs into the text and try again?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/27/23 04:12, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hello chris,
>
> thanks for adding a point and I may have missed your email.
>
> below is some of the points, which showed me how tomcat is being started in my application.
>
>
>
> * Line 19: INFO [main][2023-12-27 09:00:38,983] - org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:108) - Tomcat initialized with port(s): 9109 (http) 10109 (http)
>
> Line 21: INFO [main][2023-12-27 09:00:39,008] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting service [Tomcat]
> Line 22: INFO [main][2023-12-27 09:00:39,009] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting Servlet engine: [Apache Tomcat/9.0.75]
> Line 134: INFO [lab1workflow4scalsvc2zus1][version:1.7.99-alpha.1130][traceId:][spanId:][parentSpanId:][spanExportable:][main][2023-12-27 09:01:01,057]- org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start:Line 220 - Tomcat started on port(s): 9109 (http) 10109 (http) with context path '/lab1workflow4scalsvc2zus1
>
> *
> to launch tomcat, we are not using any command here, I am trying to build my code to an image, making some helm chart out of it, using helm chart to install in my AKS containers.
This may be the first time you mentioned Spring.
I don't know if Spring uses any of those environment variables for
anything at all.
You may need to consult the Spring documentation to see how best to set
all of those JVM parameters.
-chris
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Thursday, December 14, 2023 20:33
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/14/23 00:26, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Hai Chuck.,
>> Thanks for jumping in.,
>>
>> thats what my concern is.
>> as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
>>
>> where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
>>
>> Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
>>
>> - name: JAVA_TOOL_OPTIONS
>> value: {{ .Values.deployment.javaVMMemoryArgument }}
>>
>> - name: POD_IP
>> valueFrom:
>> fieldRef:
>> fieldPath: status.podIP
>> - name: CATALINA_OPTS
>> value: >-
>> -Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.port=49151
>> -Djava.rmi.server.hostname=$POD_IP
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.local.only=false
>> -Xms2048M -Xmx10240M
>>
>> Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
>>
>> just added to initiate a JMX process.,
>>
>> but nothing is working out.,
>> I would like to request you for further inputs
>
> How are you launching Tomcat itself? Are you using catalina.sh or
> something similar, or do you use some kind of embedded-launch?
>
> It sounds to me like using CATALINA_OPTS is causing Tomcat not to start
> at all. If you use JAVA_TOOL_OPTIONS (which Tomcat ignores), Tomcat will
> start but won't initialize JMX in the way you want it to.
>
> The solution is to look at the log files that are produced when Tomcat
> launches (or fails to launch). Are you able to observe those?
>
> It would be great to see the exact command Helm (or whatever) uses to
> attempt to launch Tomcat, and then the stdout/stderr of that attempt.
>
> The diagnostic messages shown when a JVM refuses to start are usually
> pretty good. Same thing with Tomcat, though Tomcat usually starts even
> if e.g. no connectors are successfully started because there is no
> actual requirement for Tomcat to listen to any ports.
>
> -chris
>
>> ________________________________
>> From: Chuck Caldarale <n8...@gmail.com>
>> Sent: Thursday, December 14, 2023 00:28
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>>
>>
>>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>>
>>> as you just asked .,
>>> I do have a process with Catalina.
>>>
>>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>>> Active Internet connections (only servers)
>>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>>
>>
>> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>>
>> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
>> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
>> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
>> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dtest_port=9090
>> -Dignore.endorsed.dirs=
>> -classpath
>> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
>> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
>> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
>> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
>> start
>>
>>
>> - Chuck
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello chris,
thanks for adding a point and I may have missed your email.
below is some of the points, which showed me how tomcat is being started in my application.
* Line 19: INFO [main][2023-12-27 09:00:38,983] - org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:108) - Tomcat initialized with port(s): 9109 (http) 10109 (http)
Line 21: INFO [main][2023-12-27 09:00:39,008] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting service [Tomcat]
Line 22: INFO [main][2023-12-27 09:00:39,009] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting Servlet engine: [Apache Tomcat/9.0.75]
Line 134: INFO [lab1workflow4scalsvc2zus1][version:1.7.99-alpha.1130][traceId:][spanId:][parentSpanId:][spanExportable:][main][2023-12-27 09:01:01,057]- org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start:Line 220 - Tomcat started on port(s): 9109 (http) 10109 (http) with context path '/lab1workflow4scalsvc2zus1
*
to launch tomcat, we are not using any command here, I am trying to build my code to an image, making some helm chart out of it, using helm chart to install in my AKS containers.
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, December 14, 2023 20:33
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/14/23 00:26, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hai Chuck.,
> Thanks for jumping in.,
>
> thats what my concern is.
> as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
>
> where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
>
> Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
>
> - name: JAVA_TOOL_OPTIONS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> - name: CATALINA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=49151
> -Djava.rmi.server.hostname=$POD_IP
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Xms2048M -Xmx10240M
>
> Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
>
> just added to initiate a JMX process.,
>
> but nothing is working out.,
> I would like to request you for further inputs
How are you launching Tomcat itself? Are you using catalina.sh or
something similar, or do you use some kind of embedded-launch?
It sounds to me like using CATALINA_OPTS is causing Tomcat not to start
at all. If you use JAVA_TOOL_OPTIONS (which Tomcat ignores), Tomcat will
start but won't initialize JMX in the way you want it to.
The solution is to look at the log files that are produced when Tomcat
launches (or fails to launch). Are you able to observe those?
It would be great to see the exact command Helm (or whatever) uses to
attempt to launch Tomcat, and then the stdout/stderr of that attempt.
The diagnostic messages shown when a JVM refuses to start are usually
pretty good. Same thing with Tomcat, though Tomcat usually starts even
if e.g. no connectors are successfully started because there is no
actual requirement for Tomcat to listen to any ports.
-chris
> ________________________________
> From: Chuck Caldarale <n8...@gmail.com>
> Sent: Thursday, December 14, 2023 00:28
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
>
>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>
>> as you just asked .,
>> I do have a process with Catalina.
>>
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>
>
> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>
> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dtest_port=9090
> -Dignore.endorsed.dirs=
> -classpath
> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
> start
>
>
> - Chuck
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello chris,
thanks for adding a point and I may have missed your email.
below is some of the points, which showed me how tomcat is being started in my application.
* Line 19: INFO [main][2023-12-27 09:00:38,983] - org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:108) - Tomcat initialized with port(s): 9109 (http) 10109 (http)
Line 21: INFO [main][2023-12-27 09:00:39,008] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting service [Tomcat]
Line 22: INFO [main][2023-12-27 09:00:39,009] - org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:173) - Starting Servlet engine: [Apache Tomcat/9.0.75]
Line 134: INFO [lab1workflow4scalsvc2zus1][version:1.7.99-alpha.1130][traceId:][spanId:][parentSpanId:][spanExportable:][main][2023-12-27 09:01:01,057]- org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start:Line 220 - Tomcat started on port(s): 9109 (http) 10109 (http) with context path '/lab1workflow4scalsvc2zus1
*
to launch tomcat, we are not using any command here, I am trying to build my code to an image, making some helm chart out of it, using helm chart to install in my AKS containers.
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, December 14, 2023 20:33
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/14/23 00:26, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hai Chuck.,
> Thanks for jumping in.,
>
> thats what my concern is.
> as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
>
> where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
>
> Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
>
> - name: JAVA_TOOL_OPTIONS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> - name: CATALINA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=49151
> -Djava.rmi.server.hostname=$POD_IP
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Xms2048M -Xmx10240M
>
> Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
>
> just added to initiate a JMX process.,
>
> but nothing is working out.,
> I would like to request you for further inputs
How are you launching Tomcat itself? Are you using catalina.sh or
something similar, or do you use some kind of embedded-launch?
It sounds to me like using CATALINA_OPTS is causing Tomcat not to start
at all. If you use JAVA_TOOL_OPTIONS (which Tomcat ignores), Tomcat will
start but won't initialize JMX in the way you want it to.
The solution is to look at the log files that are produced when Tomcat
launches (or fails to launch). Are you able to observe those?
It would be great to see the exact command Helm (or whatever) uses to
attempt to launch Tomcat, and then the stdout/stderr of that attempt.
The diagnostic messages shown when a JVM refuses to start are usually
pretty good. Same thing with Tomcat, though Tomcat usually starts even
if e.g. no connectors are successfully started because there is no
actual requirement for Tomcat to listen to any ports.
-chris
> ________________________________
> From: Chuck Caldarale <n8...@gmail.com>
> Sent: Thursday, December 14, 2023 00:28
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
>
>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>
>> as you just asked .,
>> I do have a process with Catalina.
>>
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>
>
> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>
> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dtest_port=9090
> -Dignore.endorsed.dirs=
> -classpath
> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
> start
>
>
> - Chuck
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/14/23 00:26, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hai Chuck.,
> Thanks for jumping in.,
>
> thats what my concern is.
> as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
>
> where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
>
> Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
>
> - name: JAVA_TOOL_OPTIONS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> - name: CATALINA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=49151
> -Djava.rmi.server.hostname=$POD_IP
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Xms2048M -Xmx10240M
>
> Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
>
> just added to initiate a JMX process.,
>
> but nothing is working out.,
> I would like to request you for further inputs
How are you launching Tomcat itself? Are you using catalina.sh or
something similar, or do you use some kind of embedded-launch?
It sounds to me like using CATALINA_OPTS is causing Tomcat not to start
at all. If you use JAVA_TOOL_OPTIONS (which Tomcat ignores), Tomcat will
start but won't initialize JMX in the way you want it to.
The solution is to look at the log files that are produced when Tomcat
launches (or fails to launch). Are you able to observe those?
It would be great to see the exact command Helm (or whatever) uses to
attempt to launch Tomcat, and then the stdout/stderr of that attempt.
The diagnostic messages shown when a JVM refuses to start are usually
pretty good. Same thing with Tomcat, though Tomcat usually starts even
if e.g. no connectors are successfully started because there is no
actual requirement for Tomcat to listen to any ports.
-chris
> ________________________________
> From: Chuck Caldarale <n8...@gmail.com>
> Sent: Thursday, December 14, 2023 00:28
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
>
>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>
>> as you just asked .,
>> I do have a process with Catalina.
>>
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>
>
> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>
> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dtest_port=9090
> -Dignore.endorsed.dirs=
> -classpath
> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
> start
>
>
> - Chuck
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by Peter Kreuser <lo...@kreuser.name>.
Sai Vamsi,
> Am 13.12.2023 um 19:59 schrieb Chuck Caldarale <n8...@gmail.com>:
>
>
>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>
>> as you just asked .,
>> I do have a process with Catalina.
>>
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>
>
you have to figure out WHY tomcat is not starting! There should be log files or error messages on the console. It seems you have put an error somewhere in any of the configfiles. It's not at all a question of the ports not being allocated. Take a step back and make tomcat launch again. After that we figure out where you have to set the options...
Please detail how you start tomcat and show the output of startup (the beginning and last lines should be enough).
Again, don't put any java options for tomcat in any global environment options (JAVA_OPTS, CATALINA_OPTS) in your shell. Only in setenv.sh .
Peter
> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>
> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dtest_port=9090
> -Dignore.endorsed.dirs=
> -classpath
> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
> start
>
>
> - Chuck
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by lo...@kreuser.name.
Sai Vamsi,
have you answered any of the questions Chris, Chuck and I asked?
Tomcat is NOT running, so please check "why" first.
We'd like to help, but can't see what is happening on your machine.
Kind regards
Peter
> Am 26.12.2023 um 12:02 schrieb Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID>:
>
> by chance any findings on this.,
>
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
> ________________________________
> From: Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com>
> Sent: Thursday, December 14, 2023 10:56
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
> Hai Chuck.,
> Thanks for jumping in.,
>
> thats what my concern is.
> as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
>
> where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
>
> Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
>
> - name: JAVA_TOOL_OPTIONS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> - name: CATALINA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=49151
> -Djava.rmi.server.hostname=$POD_IP
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Xms2048M -Xmx10240M
>
> Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
>
> just added to initiate a JMX process.,
>
> but nothing is working out.,
> I would like to request you for further inputs
>
>
> cc :chris
>
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
> ________________________________
> From: Chuck Caldarale <n8...@gmail.com>
> Sent: Thursday, December 14, 2023 00:28
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
>
>
>> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>>
>> as you just asked .,
>> I do have a process with Catalina.
>>
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::34753 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
>> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
>> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
>
>
> That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
>
> chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
> chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
> chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
> -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dtest_port=9090
> -Dignore.endorsed.dirs=
> -classpath
> /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
> -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
> -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
> start
>
>
> - Chuck
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
by chance any findings on this.,
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com>
Sent: Thursday, December 14, 2023 10:56
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
Hai Chuck.,
Thanks for jumping in.,
thats what my concern is.
as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
- name: JAVA_TOOL_OPTIONS
value: {{ .Values.deployment.javaVMMemoryArgument }}
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CATALINA_OPTS
value: >-
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=49151
-Djava.rmi.server.hostname=$POD_IP
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Xms2048M -Xmx10240M
Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
just added to initiate a JMX process.,
but nothing is working out.,
I would like to request you for further inputs
cc :chris
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Chuck Caldarale <n8...@gmail.com>
Sent: Thursday, December 14, 2023 00:28
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>
> as you just asked .,
> I do have a process with Catalina.
>
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::34753 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
-Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dtest_port=9090
-Dignore.endorsed.dirs=
-classpath
/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
-Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
start
- Chuck
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hai Chuck.,
Thanks for jumping in.,
thats what my concern is.
as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting.
where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error.
Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below
- name: JAVA_TOOL_OPTIONS
value: {{ .Values.deployment.javaVMMemoryArgument }}
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CATALINA_OPTS
value: >-
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=49151
-Djava.rmi.server.hostname=$POD_IP
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Xms2048M -Xmx10240M
Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote
just added to initiate a JMX process.,
but nothing is working out.,
I would like to request you for further inputs
cc :chris
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Chuck Caldarale <n8...@gmail.com>
Sent: Thursday, December 14, 2023 00:28
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues.
> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>
> as you just asked .,
> I do have a process with Catalina.
>
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::34753 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
-Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dtest_port=9090
-Dignore.endorsed.dirs=
-classpath
/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
-Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
start
- Chuck
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by Chuck Caldarale <n8...@gmail.com>.
> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID> wrote:
>
> as you just asked .,
> I do have a process with Catalina.
>
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::34753 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
> root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity):
chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina
chuck 16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina
chuck 16874 0.0 0.9 415316912 153296 s000 S 12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java
-Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dtest_port=9090
-Dignore.endorsed.dirs=
-classpath
/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar
-Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83
-Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap
start
- Chuck
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello Christopher.
as you just asked .,
I do have a process with Catalina.
root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::34753 :::* LISTEN 1/java
tcp6 0 0 :::9109 :::* LISTEN 1/java
tcp6 0 0 :::10109 :::* LISTEN 1/java
root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C
root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina
root 744 0.0 0.0 6460 680 pts/1 S+ 11:47 0:00 grep --color=auto catalina
root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/#
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Saturday, December 9, 2023 03:27
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/8/23 00:43, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hey Christopher.,
> Greetings of the day.
>
>
> 1.
> Might I have confused you with posting the arguments directly ., Yeah as i just shared you the annotations with comments , to state you the stuff i am using., But in my deployment ., I am using them in catalina opts., and trying to call them from values.yaml., which looks like this :
>
> javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local "
>
> and I am referring the word "javaVMMemoryArgument" from values yaml and calling it in Catalina_opts, so that it would fetch all these annotations as mentioned above, during the deployment. This is my deployment part., where I am referring to the above values from values.yaml
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
>
>
>
> 1.
> Coming to Process., I have searched for Java process that listens on my mentioned port ie., 49151, but none of the process is listening to that process.
>
> I even tried with
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep LISTEN
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | more
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':443'netstat -tulpn | grep ':443'^C
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':49151'
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
>
> which confirms me that , any of the process is being listening on the port 49151.
>
> 2.
> I would like to request you to suggest me with a better approach ., where i am missing anything in this process!
Good question. What is pid #1? Do those port numbers make any sense for
your Tomcat-based service?
Is Tomcat even running? Try 'ps aux | grep catalina' to see if there are
any. Are you launching Tomcat using catalina.sh / startup.sh or similar?
Or are you running Tomcat "embedded" within your own application?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/8/23 00:43, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hey Christopher.,
> Greetings of the day.
>
>
> 1.
> Might I have confused you with posting the arguments directly ., Yeah as i just shared you the annotations with comments , to state you the stuff i am using., But in my deployment ., I am using them in catalina opts., and trying to call them from values.yaml., which looks like this :
>
> javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local "
>
> and I am referring the word "javaVMMemoryArgument" from values yaml and calling it in Catalina_opts, so that it would fetch all these annotations as mentioned above, during the deployment. This is my deployment part., where I am referring to the above values from values.yaml
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
>
>
>
> 1.
> Coming to Process., I have searched for Java process that listens on my mentioned port ie., 49151, but none of the process is listening to that process.
>
> I even tried with
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep LISTEN
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | more
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':443'netstat -tulpn | grep ':443'^C
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':49151'
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
>
> which confirms me that , any of the process is being listening on the port 49151.
>
> 2.
> I would like to request you to suggest me with a better approach ., where i am missing anything in this process!
Good question. What is pid #1? Do those port numbers make any sense for
your Tomcat-based service?
Is Tomcat even running? Try 'ps aux | grep catalina' to see if there are
any. Are you launching Tomcat using catalina.sh / startup.sh or similar?
Or are you running Tomcat "embedded" within your own application?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Posted by lo...@kreuser.name.
Sai Vamsi,
> Am 08.12.2023 um 06:43 schrieb Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID>:
>
> Hey Christopher.,
> Greetings of the day.
>
>
> 1.
> Might I have confused you with posting the arguments directly ., Yeah as i just shared you the annotations with comments , to state you the stuff i am using., But in my deployment ., I am using them in catalina opts., and trying to call them from values.yaml., which looks like this :
>
> javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local "
>
> and I am referring the word "javaVMMemoryArgument" from values yaml and calling it in Catalina_opts, so that it would fetch all these annotations as mentioned above, during the deployment. This is my deployment part., where I am referring to the above values from values.yaml
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
I compare with my CATALINA_OPTS (and I noticed that some of my options were still in JAVA_OPTS boooh!) :
export CATALINA_OPTS="${CATALINA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${HOSTNAME} -Dcom.sun.management.jmxremote.local.only=false"
I see all my options in
root@tomcat:/usr/local/tomcat# ps -aef | less
UID PID PPID C STIME TTY TIME CMD
tomcat 1 0 71 10:42 ? 00:04:02 /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat.base/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -agentlib:jdwp=transport=dt_socket,address=0.0.0.0:8000,server=y,suspend=n -XX:NativeMemoryTracking=summary -Dhostname=<hostname> -Djava.awt.headless=true -Djavax.net.ssl.trustStore=/opt/apache-tomcat.base/conf/ssl/cacerts.jks -Xlog:gc:/opt/apache-tomcat.base/logs/gc.log -Djava.security.egd=file:/dev/urandom -Dsun.net.inetaddr.ttl=60 -Djava.library.path=/usr/local/tomcat/native-jni-lib -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdk.tls.server.enableStatusRequestExtension=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<hostname> -Dcom.sun.management.jmxremote.local.only=false -javaagent:/opt/apache-tomcat.base/bin/jmx_prometheus_javaagent-0.12.0.jar=8080:/opt/apache-tomcat.base/bin/tomcat.yaml -Djava.net.debug=ssl -XX:+UnlockDiagnosticVMOptions -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/apache-tomcat.base -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/opt/apache-tomcat.base/temp org.apache.catalina.startup.Bootstrap start
Ports are opened
root@tomcat:/usr/local/tomcat# netstat -tulpn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
...
tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN -
...
tcp 2 0 0.0.0.0:8443 0.0.0.0:* LISTEN -
...
jconsole <hostname>:10001
> JMX connect success
>
>
>
> 1.
> Coming to Process., I have searched for Java process that listens on my mentioned port ie., 49151, but none of the process is listening to that process.
>
> I even tried with
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep LISTEN
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | more
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':443'netstat -tulpn | grep ':443'^C
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':49151'
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
>
> which confirms me that , any of the process is being listening on the port 49151.
>
You should at least see some error when starting Tomcat. Anything in the logs or on the console?
Peter
> 2.
> I would like to request you to suggest me with a better approach ., where i am missing anything in this process!
>
>
>
>
>
>
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Thursday, December 7, 2023 23:33
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Hello Christopher.
>> Thanks for the response!
>>
>> Let me walk you through the Environment.,
>> I have a cluster and I am installing multiple microservices in the cluster. And one of the microservices I am integrating the JMX on of the microservices.,
>> and the mentioned IP of the pod is being changed dynamically after each deployment .
>>
>> Here are the annotations that I am using in my deployment:
>> - name: JAVA_OPTS
>> # value: >-
>> # -Dcom.sun.management.jmxremote
>> # -Dcom.sun.management.jmxremote.authenticate=false
>> # -Dcom.sun.management.jmxremote.ssl=false
>> # -Dcom.sun.management.jmxremote.local.only=false
>> # -Dcom.sun.management.jmxremote.port=49151
>> # -Dcom.sun.management.jmxremote.rmi.port=49151
>> # -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
>
> If that's YAML, then all of the above is comments. Those lines do nothing.
>
>> as i said , I have implemented these in three scenarios., Catalina_opts / Java_opts / Java_Tool_opts
>
> You should definitely use CATALINA_OPTS and none of the other things you
> mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
> don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
> Tomcat-related scripts, and you'll find that even "shutdown" causes an
> error... because it's trying to initialize JMX when shutting-down, which
> is again useless.
>
>> I have tried deploying catalina_ots and Java_opts as in the case of above mentioned.
>> I also tried other way, mentioning all these in the values.yaml file ( I am using Helm Deployment to upgrade my deployments) and referring these annotations as , and mentioning them in the Value
>> env:
>> - name: CATALINA_OPTS
>> value: {{ .Values.deployment.javaVMMemoryArgument }}
>>
>>
>> getting netstat output from my pod:
>> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>>
>> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep tomcat
>> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep --color=auto tomcat
>
> Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
> 'tomcat' does not appear in a normal JVM launch command for Tomcat.
>
>> right now , I am passing my args with Catalina_opts in my deployment
>> and I am not able to get any tomcat process pid while i check with
>> netstat -apn , I can see multiple Java Process but not single tomcat process
>>
>> My Jmx port is being configured as a service., so that it could be
>> bounded with Tcp protocol externally to the container., and allow all
>> tcp connections.
>>
>> Hope I am able to provide you some detailed information., Please let
>> me know., if I can help you with anything in further!
>
> I think maybe you are both configuring your YAML file incorrectly and
> also looking for the process incorrectly.
>
> -chris
>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Wednesday, December 6, 2023 20:40
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsi,
>>
>> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
>>> It came back to previous stage where Annotations are not on the JVM .,
>>> Came back to previous stage, where connection refused error., as follows.
>>>
>>> Could not initialize instance: tomcat-10.75.12.136-49151:
>>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
>>> java.net.ConnectException: Connection refused (Connection refused)]
>>> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
>>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
>>> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
>>> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
>>> at org.datadog.jmxfetch.App.init(App.java:917)
>>> at org.datadog.jmxfetch.App.start(App.java:413)
>>> at org.datadog.jmxfetch.App.run(App.java:173)
>>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
>>> at org.datadog.jmxfetch.App.main(App.java:91)
>>
>> Right: the DataDog server is unable to connect to your appication via
>> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
>> and that DataDog isn't running on that server. It's on some other IP, right?
>>
>>> Here we were not able to find any process , to confirm that atleast
>>> port is being listened to.
>> What is your exact CATALINA_OPTS value (masking any secrets that may be
>> in there).
>>
>> Is Tomcat definitely running? Can you prove that with a "ps" listing,
>> for example? That will also confirm it's running with the proper
>> CATALINA_OPTS settings.
>>
>> Can you run "netstat" and show all of the ports your Tomcat process is
>> listening to?
>>
>> I'm assuming that your Tomcat process is being hosted in some kind of
>> container. Is the IP address 10.75.12.136 the IP of the container, or
>> the IP of the host which hosts the container? Is your JMX port
>> configured to be visible external to the container itself, or only
>> internally to the container?
>>
>> -chris
>>
>>
>>
>>> Thanks & Regards,
>>> --
>>>
>>> SAI VAMSI .B
>>> Senior DevOps Engineer
>>>
>>> ________________________________
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Sent: Tuesday, December 5, 2023 19:19
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>
>>> Sai Vamsa,
>>>
>>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>> Thanks for the response..
>>>>
>>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>>>
>>>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>>>> Let me know if I am missing anything.
>>>
>>> Okay. Can you please summarize the current situation? I think you've
>>> made a few changes, etc. so it would help to know the current state.
>>>
>>> Stop Tomcat and any other related services, delete all appropriate log
>>> files, etc. and start Tomcat (or attempt to). Then post any output that
>>> is relevant, and if you get things like "address in use" then please
>>> post the details about which process id owns that port and the full
>>> command-line that launched that process.
>>>
>>> Thanks,
>>> -chris
>>>
>>>> ________________________________
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Sent: Monday, December 4, 2023 23:22
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>>
>>>> Sai Vamsi,
>>>>
>>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>>>
>>>> You will probably want to use CATALINA_OPTS instead of any of the other
>>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>>>> need to have the JMX subsystem starting-up and trying to grab a port.
>>>>
>>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>>>> processes, like jps for example...
>>>>
>>>>> yeah they are reflecting and creating a Java Process.
>>>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>>>
>>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>>>> java.net.BindException: Address already in use (Bind failed)
>>>>
>>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>>>> to your port which is aready bound by your Tomcat process.
>>>>
>>>> The solution is to use only CATALINA_OPTS to set these options.
>>>>
>>>> -chris
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hey Christopher.,
Greetings of the day.
1.
Might I have confused you with posting the arguments directly ., Yeah as i just shared you the annotations with comments , to state you the stuff i am using., But in my deployment ., I am using them in catalina opts., and trying to call them from values.yaml., which looks like this :
javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local "
and I am referring the word "javaVMMemoryArgument" from values yaml and calling it in Catalina_opts, so that it would fetch all these annotations as mentioned above, during the deployment. This is my deployment part., where I am referring to the above values from values.yaml
env:
- name: CATALINA_OPTS
value: {{ .Values.deployment.javaVMMemoryArgument }}
1.
Coming to Process., I have searched for Java process that listens on my mentioned port ie., 49151, but none of the process is listening to that process.
I even tried with
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep LISTEN
tcp6 0 0 :::10109 :::* LISTEN 1/java
tcp6 0 0 :::9109 :::* LISTEN 1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | more
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::10109 :::* LISTEN 1/java
tcp6 0 0 :::9109 :::* LISTEN 1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':443'netstat -tulpn | grep ':443'^C
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':49151'
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
which confirms me that , any of the process is being listening on the port 49151.
2.
I would like to request you to suggest me with a better approach ., where i am missing anything in this process!
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, December 7, 2023 23:33
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hello Christopher.
> Thanks for the response!
>
> Let me walk you through the Environment.,
> I have a cluster and I am installing multiple microservices in the cluster. And one of the microservices I am integrating the JMX on of the microservices.,
> and the mentioned IP of the pod is being changed dynamically after each deployment .
>
> Here are the annotations that I am using in my deployment:
> - name: JAVA_OPTS
> # value: >-
> # -Dcom.sun.management.jmxremote
> # -Dcom.sun.management.jmxremote.authenticate=false
> # -Dcom.sun.management.jmxremote.ssl=false
> # -Dcom.sun.management.jmxremote.local.only=false
> # -Dcom.sun.management.jmxremote.port=49151
> # -Dcom.sun.management.jmxremote.rmi.port=49151
> # -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
If that's YAML, then all of the above is comments. Those lines do nothing.
> as i said , I have implemented these in three scenarios., Catalina_opts / Java_opts / Java_Tool_opts
You should definitely use CATALINA_OPTS and none of the other things you
mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
Tomcat-related scripts, and you'll find that even "shutdown" causes an
error... because it's trying to initialize JMX when shutting-down, which
is again useless.
> I have tried deploying catalina_ots and Java_opts as in the case of above mentioned.
> I also tried other way, mentioning all these in the values.yaml file ( I am using Helm Deployment to upgrade my deployments) and referring these annotations as , and mentioning them in the Value
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
> getting netstat output from my pod:
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
>
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep tomcat
> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep --color=auto tomcat
Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
'tomcat' does not appear in a normal JVM launch command for Tomcat.
> right now , I am passing my args with Catalina_opts in my deployment
> and I am not able to get any tomcat process pid while i check with
> netstat -apn , I can see multiple Java Process but not single tomcat process
>
> My Jmx port is being configured as a service., so that it could be
> bounded with Tcp protocol externally to the container., and allow all
> tcp connections.
>
> Hope I am able to provide you some detailed information., Please let
> me know., if I can help you with anything in further!
I think maybe you are both configuring your YAML file incorrectly and
also looking for the process incorrectly.
-chris
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, December 6, 2023 20:40
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
>> It came back to previous stage where Annotations are not on the JVM .,
>> Came back to previous stage, where connection refused error., as follows.
>>
>> Could not initialize instance: tomcat-10.75.12.136-49151:
>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
>> java.net.ConnectException: Connection refused (Connection refused)]
>> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
>> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
>> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
>> at org.datadog.jmxfetch.App.init(App.java:917)
>> at org.datadog.jmxfetch.App.start(App.java:413)
>> at org.datadog.jmxfetch.App.run(App.java:173)
>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
>> at org.datadog.jmxfetch.App.main(App.java:91)
>
> Right: the DataDog server is unable to connect to your appication via
> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
> and that DataDog isn't running on that server. It's on some other IP, right?
>
>> Here we were not able to find any process , to confirm that atleast
>> port is being listened to.
> What is your exact CATALINA_OPTS value (masking any secrets that may be
> in there).
>
> Is Tomcat definitely running? Can you prove that with a "ps" listing,
> for example? That will also confirm it's running with the proper
> CATALINA_OPTS settings.
>
> Can you run "netstat" and show all of the ports your Tomcat process is
> listening to?
>
> I'm assuming that your Tomcat process is being hosted in some kind of
> container. Is the IP address 10.75.12.136 the IP of the container, or
> the IP of the host which hosts the container? Is your JMX port
> configured to be visible external to the container itself, or only
> internally to the container?
>
> -chris
>
>
>
>> Thanks & Regards,
>> --
>>
>> SAI VAMSI .B
>> Senior DevOps Engineer
>>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Tuesday, December 5, 2023 19:19
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsa,
>>
>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> Thanks for the response..
>>>
>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>>
>>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>>> Let me know if I am missing anything.
>>
>> Okay. Can you please summarize the current situation? I think you've
>> made a few changes, etc. so it would help to know the current state.
>>
>> Stop Tomcat and any other related services, delete all appropriate log
>> files, etc. and start Tomcat (or attempt to). Then post any output that
>> is relevant, and if you get things like "address in use" then please
>> post the details about which process id owns that port and the full
>> command-line that launched that process.
>>
>> Thanks,
>> -chris
>>
>>> ________________________________
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Sent: Monday, December 4, 2023 23:22
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>
>>> Sai Vamsi,
>>>
>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>>
>>> You will probably want to use CATALINA_OPTS instead of any of the other
>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>>> need to have the JMX subsystem starting-up and trying to grab a port.
>>>
>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>>> processes, like jps for example...
>>>
>>>> yeah they are reflecting and creating a Java Process.
>>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>>
>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>>> java.net.BindException: Address already in use (Bind failed)
>>>
>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>>> to your port which is aready bound by your Tomcat process.
>>>
>>> The solution is to use only CATALINA_OPTS to set these options.
>>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hello Christopher.
> Thanks for the response!
>
> Let me walk you through the Environment.,
> I have a cluster and I am installing multiple microservices in the cluster. And one of the microservices I am integrating the JMX on of the microservices.,
> and the mentioned IP of the pod is being changed dynamically after each deployment .
>
> Here are the annotations that I am using in my deployment:
> - name: JAVA_OPTS
> # value: >-
> # -Dcom.sun.management.jmxremote
> # -Dcom.sun.management.jmxremote.authenticate=false
> # -Dcom.sun.management.jmxremote.ssl=false
> # -Dcom.sun.management.jmxremote.local.only=false
> # -Dcom.sun.management.jmxremote.port=49151
> # -Dcom.sun.management.jmxremote.rmi.port=49151
> # -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
If that's YAML, then all of the above is comments. Those lines do nothing.
> as i said , I have implemented these in three scenarios., Catalina_opts / Java_opts / Java_Tool_opts
You should definitely use CATALINA_OPTS and none of the other things you
mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
Tomcat-related scripts, and you'll find that even "shutdown" causes an
error... because it's trying to initialize JMX when shutting-down, which
is again useless.
> I have tried deploying catalina_ots and Java_opts as in the case of above mentioned.
> I also tried other way, mentioning all these in the values.yaml file ( I am using Helm Deployment to upgrade my deployments) and referring these annotations as , and mentioning them in the Value
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
> getting netstat output from my pod:
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
>
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep tomcat
> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep --color=auto tomcat
Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
'tomcat' does not appear in a normal JVM launch command for Tomcat.
> right now , I am passing my args with Catalina_opts in my deployment
> and I am not able to get any tomcat process pid while i check with
> netstat -apn , I can see multiple Java Process but not single tomcat process
>
> My Jmx port is being configured as a service., so that it could be
> bounded with Tcp protocol externally to the container., and allow all
> tcp connections.
>
> Hope I am able to provide you some detailed information., Please let
> me know., if I can help you with anything in further!
I think maybe you are both configuring your YAML file incorrectly and
also looking for the process incorrectly.
-chris
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, December 6, 2023 20:40
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
>> It came back to previous stage where Annotations are not on the JVM .,
>> Came back to previous stage, where connection refused error., as follows.
>>
>> Could not initialize instance: tomcat-10.75.12.136-49151:
>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
>> java.net.ConnectException: Connection refused (Connection refused)]
>> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
>> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
>> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
>> at org.datadog.jmxfetch.App.init(App.java:917)
>> at org.datadog.jmxfetch.App.start(App.java:413)
>> at org.datadog.jmxfetch.App.run(App.java:173)
>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
>> at org.datadog.jmxfetch.App.main(App.java:91)
>
> Right: the DataDog server is unable to connect to your appication via
> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
> and that DataDog isn't running on that server. It's on some other IP, right?
>
>> Here we were not able to find any process , to confirm that atleast
>> port is being listened to.
> What is your exact CATALINA_OPTS value (masking any secrets that may be
> in there).
>
> Is Tomcat definitely running? Can you prove that with a "ps" listing,
> for example? That will also confirm it's running with the proper
> CATALINA_OPTS settings.
>
> Can you run "netstat" and show all of the ports your Tomcat process is
> listening to?
>
> I'm assuming that your Tomcat process is being hosted in some kind of
> container. Is the IP address 10.75.12.136 the IP of the container, or
> the IP of the host which hosts the container? Is your JMX port
> configured to be visible external to the container itself, or only
> internally to the container?
>
> -chris
>
>
>
>> Thanks & Regards,
>> --
>>
>> SAI VAMSI .B
>> Senior DevOps Engineer
>>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Tuesday, December 5, 2023 19:19
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsa,
>>
>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> Thanks for the response..
>>>
>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>>
>>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>>> Let me know if I am missing anything.
>>
>> Okay. Can you please summarize the current situation? I think you've
>> made a few changes, etc. so it would help to know the current state.
>>
>> Stop Tomcat and any other related services, delete all appropriate log
>> files, etc. and start Tomcat (or attempt to). Then post any output that
>> is relevant, and if you get things like "address in use" then please
>> post the details about which process id owns that port and the full
>> command-line that launched that process.
>>
>> Thanks,
>> -chris
>>
>>> ________________________________
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Sent: Monday, December 4, 2023 23:22
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>
>>> Sai Vamsi,
>>>
>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>>
>>> You will probably want to use CATALINA_OPTS instead of any of the other
>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>>> need to have the JMX subsystem starting-up and trying to grab a port.
>>>
>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>>> processes, like jps for example...
>>>
>>>> yeah they are reflecting and creating a Java Process.
>>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>>
>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>>> java.net.BindException: Address already in use (Bind failed)
>>>
>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>>> to your port which is aready bound by your Tomcat process.
>>>
>>> The solution is to use only CATALINA_OPTS to set these options.
>>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello Christopher.
Thanks for the response!
Let me walk you through the Environment.,
I have a cluster and I am installing multiple microservices in the cluster. And one of the microservices I am integrating the JMX on of the microservices.,
and the mentioned IP of the pod is being changed dynamically after each deployment .
Here are the annotations that I am using in my deployment:
- name: JAVA_OPTS
# value: >-
# -Dcom.sun.management.jmxremote
# -Dcom.sun.management.jmxremote.authenticate=false
# -Dcom.sun.management.jmxremote.ssl=false
# -Dcom.sun.management.jmxremote.local.only=false
# -Dcom.sun.management.jmxremote.port=49151
# -Dcom.sun.management.jmxremote.rmi.port=49151
# -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
as i said , I have implemented these in three scenarios., Catalina_opts / Java_opts / Java_Tool_opts
I have tried deploying catalina_ots and Java_opts as in the case of above mentioned.
I also tried other way, mentioning all these in the values.yaml file ( I am using Helm Deployment to upgrade my deployments) and referring these annotations as , and mentioning them in the Value
env:
- name: CATALINA_OPTS
value: {{ .Values.deployment.javaVMMemoryArgument }}
getting netstat output from my pod:
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::10109 :::* LISTEN 1/java
tcp6 0 0 :::9109 :::* LISTEN 1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep tomcat
root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep --color=auto tomcat
right now , I am passing my args with Catalina_opts in my deployment and I am not able to get any tomcat process pid while i check with netstat -apn , I can see multiple Java Process but not single tomcat process
My Jmx port is being configured as a service., so that it could be bounded with Tcp protocol externally to the container., and allow all tcp connections.
Hope I am able to provide you some detailed information.,
Please let me know., if I can help you with anything in further!
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, December 6, 2023 20:40
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
> It came back to previous stage where Annotations are not on the JVM .,
> Came back to previous stage, where connection refused error., as follows.
>
> Could not initialize instance: tomcat-10.75.12.136-49151:
> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
> at org.datadog.jmxfetch.App.init(App.java:917)
> at org.datadog.jmxfetch.App.start(App.java:413)
> at org.datadog.jmxfetch.App.run(App.java:173)
> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
> at org.datadog.jmxfetch.App.main(App.java:91)
Right: the DataDog server is unable to connect to your appication via
JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
and that DataDog isn't running on that server. It's on some other IP, right?
> Here we were not able to find any process , to confirm that atleast
> port is being listened to.
What is your exact CATALINA_OPTS value (masking any secrets that may be
in there).
Is Tomcat definitely running? Can you prove that with a "ps" listing,
for example? That will also confirm it's running with the proper
CATALINA_OPTS settings.
Can you run "netstat" and show all of the ports your Tomcat process is
listening to?
I'm assuming that your Tomcat process is being hosted in some kind of
container. Is the IP address 10.75.12.136 the IP of the container, or
the IP of the host which hosts the container? Is your JMX port
configured to be visible external to the container itself, or only
internally to the container?
-chris
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Tuesday, December 5, 2023 19:19
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsa,
>
> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Thanks for the response..
>>
>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>
>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>> Let me know if I am missing anything.
>
> Okay. Can you please summarize the current situation? I think you've
> made a few changes, etc. so it would help to know the current state.
>
> Stop Tomcat and any other related services, delete all appropriate log
> files, etc. and start Tomcat (or attempt to). Then post any output that
> is relevant, and if you get things like "address in use" then please
> post the details about which process id owns that port and the full
> command-line that launched that process.
>
> Thanks,
> -chris
>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Monday, December 4, 2023 23:22
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsi,
>>
>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>
>> You will probably want to use CATALINA_OPTS instead of any of the other
>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>> need to have the JMX subsystem starting-up and trying to grab a port.
>>
>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>> processes, like jps for example...
>>
>>> yeah they are reflecting and creating a Java Process.
>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>
>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>> java.net.BindException: Address already in use (Bind failed)
>>
>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>> to your port which is aready bound by your Tomcat process.
>>
>> The solution is to use only CATALINA_OPTS to set these options.
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
> It came back to previous stage where Annotations are not on the JVM .,
> Came back to previous stage, where connection refused error., as follows.
>
> Could not initialize instance: tomcat-10.75.12.136-49151:
> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
> at org.datadog.jmxfetch.App.init(App.java:917)
> at org.datadog.jmxfetch.App.start(App.java:413)
> at org.datadog.jmxfetch.App.run(App.java:173)
> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
> at org.datadog.jmxfetch.App.main(App.java:91)
Right: the DataDog server is unable to connect to your appication via
JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
and that DataDog isn't running on that server. It's on some other IP, right?
> Here we were not able to find any process , to confirm that atleast
> port is being listened to.
What is your exact CATALINA_OPTS value (masking any secrets that may be
in there).
Is Tomcat definitely running? Can you prove that with a "ps" listing,
for example? That will also confirm it's running with the proper
CATALINA_OPTS settings.
Can you run "netstat" and show all of the ports your Tomcat process is
listening to?
I'm assuming that your Tomcat process is being hosted in some kind of
container. Is the IP address 10.75.12.136 the IP of the container, or
the IP of the host which hosts the container? Is your JMX port
configured to be visible external to the container itself, or only
internally to the container?
-chris
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Tuesday, December 5, 2023 19:19
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsa,
>
> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Thanks for the response..
>>
>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>
>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>> Let me know if I am missing anything.
>
> Okay. Can you please summarize the current situation? I think you've
> made a few changes, etc. so it would help to know the current state.
>
> Stop Tomcat and any other related services, delete all appropriate log
> files, etc. and start Tomcat (or attempt to). Then post any output that
> is relevant, and if you get things like "address in use" then please
> post the details about which process id owns that port and the full
> command-line that launched that process.
>
> Thanks,
> -chris
>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Monday, December 4, 2023 23:22
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsi,
>>
>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>
>> You will probably want to use CATALINA_OPTS instead of any of the other
>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>> need to have the JMX subsystem starting-up and trying to grab a port.
>>
>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>> processes, like jps for example...
>>
>>> yeah they are reflecting and creating a Java Process.
>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>
>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>> java.net.BindException: Address already in use (Bind failed)
>>
>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>> to your port which is aready bound by your Tomcat process.
>>
>> The solution is to use only CATALINA_OPTS to set these options.
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
As you suggested.,
I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
It came back to previous stage where Annotations are not on the JVM .,
Came back to previous stage, where connection refused error., as follows.
Could not initialize instance: tomcat-10.75.12.136-49151:
java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
at org.datadog.jmxfetch.App$5.invoke(App.java:927)
at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
at org.datadog.jmxfetch.App.init(App.java:917)
at org.datadog.jmxfetch.App.start(App.java:413)
at org.datadog.jmxfetch.App.run(App.java:173)
at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
at org.datadog.jmxfetch.App.main(App.java:91)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.datadog.jmxfetch.Connection.createConnection(Connection.java:63)
at org.datadog.jmxfetch.RemoteConnection.<init>(RemoteConnection.java:101)
at org.datadog.jmxfetch.ConnectionFactory.createConnection(ConnectionFactory.java:38)
at org.datadog.jmxfetch.Instance.getConnection(Instance.java:405)
at org.datadog.jmxfetch.Instance.init(Instance.java:418)
at org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:15)
at org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:3)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)
at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
... 12 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
... 17 more
Here we were not able to find any process , to confirm that atleast port is being listened to.
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Tuesday, December 5, 2023 19:19
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsa,
On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Thanks for the response..
>
> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>
> But its not even working out., Java process is not getting created., and port is not bound to any process.
> Let me know if I am missing anything.
Okay. Can you please summarize the current situation? I think you've
made a few changes, etc. so it would help to know the current state.
Stop Tomcat and any other related services, delete all appropriate log
files, etc. and start Tomcat (or attempt to). Then post any output that
is relevant, and if you get things like "address in use" then please
post the details about which process id owns that port and the full
command-line that launched that process.
Thanks,
-chris
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, December 4, 2023 23:22
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>
> You will probably want to use CATALINA_OPTS instead of any of the other
> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
> any time it invokes a JVM. For example, running bind/digest.sh doesn't
> need to have the JMX subsystem starting-up and trying to grab a port.
>
> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
> processes, like jps for example...
>
>> yeah they are reflecting and creating a Java Process.
>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>
>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>> java.net.BindException: Address already in use (Bind failed)
>
> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
> to your port which is aready bound by your Tomcat process.
>
> The solution is to use only CATALINA_OPTS to set these options.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsa,
On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Thanks for the response..
>
> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>
> But its not even working out., Java process is not getting created., and port is not bound to any process.
> Let me know if I am missing anything.
Okay. Can you please summarize the current situation? I think you've
made a few changes, etc. so it would help to know the current state.
Stop Tomcat and any other related services, delete all appropriate log
files, etc. and start Tomcat (or attempt to). Then post any output that
is relevant, and if you get things like "address in use" then please
post the details about which process id owns that port and the full
command-line that launched that process.
Thanks,
-chris
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, December 4, 2023 23:22
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>
> You will probably want to use CATALINA_OPTS instead of any of the other
> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
> any time it invokes a JVM. For example, running bind/digest.sh doesn't
> need to have the JMX subsystem starting-up and trying to grab a port.
>
> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
> processes, like jps for example...
>
>> yeah they are reflecting and creating a Java Process.
>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>
>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>> java.net.BindException: Address already in use (Bind failed)
>
> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
> to your port which is aready bound by your Tomcat process.
>
> The solution is to use only CATALINA_OPTS to set these options.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello Chris.
Thanks for the response..
as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
But its not even working out., Java process is not getting created., and port is not bound to any process.
Let me know if I am missing anything.
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, December 4, 2023 23:22
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
You will probably want to use CATALINA_OPTS instead of any of the other
ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
any time it invokes a JVM. For example, running bind/digest.sh doesn't
need to have the JMX subsystem starting-up and trying to grab a port.
JAVA_TOOL_OPTS is an environment variable used by JVM-launching
processes, like jps for example...
> yeah they are reflecting and creating a Java Process.
> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>
> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
> java.net.BindException: Address already in use (Bind failed)
Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
to your port which is aready bound by your Tomcat process.
The solution is to use only CATALINA_OPTS to set these options.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hey Thomas,
Thanks for the response.
Yeah., I am trying to add Heap size parameters to the JVM using the annotations.,
DO you smell anything wrong in this!?
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Thomas Meyer <th...@m3y3r.de>
Sent: Tuesday, December 5, 2023 00:03
To: Tomcat Users List <us...@tomcat.apache.org>; Christopher Schultz <ch...@christopherschultz.net>; users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Hi,
Also newer jvm do have XX:+UseContainerSupport set as default, also when using XX:+UseContainerSupport setting -Xms and -Xmx do not really make sense at all, you either want the JVM to deduce the max heap size from the memory cgroup or not....
Mfg
Thomas
Am 4. Dezember 2023 18:52:13 MEZ schrieb Christopher Schultz <ch...@christopherschultz.net>:
>Sai Vamsi,
>
>On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>
>You will probably want to use CATALINA_OPTS instead of any of the other ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat. You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS any time it invokes a JVM. For example, running bind/digest.sh doesn't need to have the JMX subsystem starting-up and trying to grab a port.
>
>JAVA_TOOL_OPTS is an environment variable used by JVM-launching processes, like jps for example...
>
>> yeah they are reflecting and creating a Java Process.
>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>
>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>> java.net.BindException: Address already in use (Bind failed)
>
>Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind to your port which is aready bound by your Tomcat process.
>
>The solution is to use only CATALINA_OPTS to set these options.
>
>-chris
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Thomas Meyer <th...@m3y3r.de>.
Hi,
Also newer jvm do have XX:+UseContainerSupport set as default, also when using XX:+UseContainerSupport setting -Xms and -Xmx do not really make sense at all, you either want the JVM to deduce the max heap size from the memory cgroup or not....
Mfg
Thomas
Am 4. Dezember 2023 18:52:13 MEZ schrieb Christopher Schultz <ch...@christopherschultz.net>:
>Sai Vamsi,
>
>On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>
>You will probably want to use CATALINA_OPTS instead of any of the other ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat. You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS any time it invokes a JVM. For example, running bind/digest.sh doesn't need to have the JMX subsystem starting-up and trying to grab a port.
>
>JAVA_TOOL_OPTS is an environment variable used by JVM-launching processes, like jps for example...
>
>> yeah they are reflecting and creating a Java Process.
>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>
>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>> java.net.BindException: Address already in use (Bind failed)
>
>Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind to your port which is aready bound by your Tomcat process.
>
>The solution is to use only CATALINA_OPTS to set these options.
>
>-chris
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
You will probably want to use CATALINA_OPTS instead of any of the other
ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
any time it invokes a JVM. For example, running bind/digest.sh doesn't
need to have the JMX subsystem starting-up and trying to grab a port.
JAVA_TOOL_OPTS is an environment variable used by JVM-launching
processes, like jps for example...
> yeah they are reflecting and creating a Java Process.
> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>
> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
> java.net.BindException: Address already in use (Bind failed)
Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
to your port which is aready bound by your Tomcat process.
The solution is to use only CATALINA_OPTS to set these options.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by "Bodavula, Sai Vamsi Mohan Krishna (TR Technology)" <Sa...@thomsonreuters.com.INVALID>.
Hello Christopher.,
Thanks for the Response.
Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
yeah they are reflecting and creating a Java Process.
but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:491)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
Caused by: java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
at java.rmi/sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:220)
at java.rmi/sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:180)
at jdk.management.agent/sun.management.jmxremote.SingleEntryRegistry.<init>(SingleEntryRegistry.java:49)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:836)
at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
... 2 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:452)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:395)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:257)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:149)
at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:670)
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324)
... 11 more
------------------------------------------------------------------------
Java process is been referred as Default JAva process:
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# ps -aux | grep java
root 1 5.3 1.9 20332668 2525004 ? Ssl 07:42 3:41 java -jar /app/workflow-service.jar
root 928 0.0 0.0 6328 684 pts/0 S+ 08:50 0:00 grep --color=auto java
and port usage is as :
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::10109 :::* LISTEN 1/java
tcp6 0 0 :::9109 :::* LISTEN 1/java
tcp6 0 0 :::49151 :::* LISTEN 1/java
tcp6 0 0 :::42965 :::* LISTEN 1/java
I have used an extreme port range, to make sure that the concerned port should not be in listening list on the application end.
Please let me know if you have any suggestions me to refer from my end!
Thanks
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, November 30, 2023 04:07
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 11/29/23 03:50, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> I am trying to add Tomcat-JMX Integration to the Datadog Agent, in order to achieve Remote Monitoring .
> I am following the docs https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$
> [https://urldefense.com/v3/__https://datadog-docs.imgix.net/images/dd-docs-meta-image.png__;!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPnR3EJUUg$ ]<https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$ >
> Autodiscovery with JMX<https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$ >
> Datadog, the leading service for cloud-scale monitoring.
> docs.datadoghq.com
>
>
> This is the Environment I am using in my case.
> I am trying to integrate JMX in one my Deployment, in AKS environment and I am using Helm to upgrade my environment.
> as suggested in the documentation., I am adding the annotations in my deployment file in helm structure.
Okay, I know absolutely nothing about Azure, Kubernetes, or Helm. But it
looks like maybe this is more basic than that...
> My agent is JMX enabled, and i can see in deployment logs.
> I have configured a service for this deployment and its reflecting.
>
>
> Issue I am facing :
> I am trying to add my annotations, in the deployment file, after deployment, it should be able make the remote connection , but its showing these logs in the agent.
I will be of no help in figuring out what "annotations" you need in your
"deployment file" because - again - I know nothing about Azure, Kube,
and Helm.
> while i check the Java process., even its not reflecting this JMX process , it seems JMX process is not at all created, in my case.,
>
> error i am facing after deployment"
> ========
> JMXFetch
> ========
>
> Information
> ==================
> runtime_version : 11.0.18
> version : 0.47.8
> Initialized checks
> ==================
> tomcat
> instance_name : tomcat-10.75.12.143-1099
> message : Unable to instantiate or initialize instance 10.75.12.143:1099. Is the target JMX Server or JVM running? Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.143; nested exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
Well THAT seems straightforward.
First: is that the right IP address?
Second: on that server (10.75.12.143), is port 1099 bound-to by the
Tomcat process? Can you, for example, nc to that TCP/IP port? Have you
checked the Tomcat logs to see if there any errors during startup... for
example, any JMX-related errors?
Third: on that IP, is port 1099 reachable from the Helm (or whatever)
server that is trying to contact it? Try using "nc 10.75.12.143 1099"
and see if you can connect from Helm (or whatever).
> - name: JAVA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.port=1099
> -Dcom.sun.management.jmxremote.rmi.port=1099
> -Djava.rmi.server.hostname=$(POD_IP)
> -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.host=$(POD_IP)
Can you login to 10.75.12.143 and check to see if the JVM launch command
has (a) all of the above -D arguments in it and (b) if the host/hostname
have the expected values in them?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Sai Vamsi,
On 11/29/23 03:50, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> I am trying to add Tomcat-JMX Integration to the Datadog Agent, in order to achieve Remote Monitoring .
> I am following the docs https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations
> [https://datadog-docs.imgix.net/images/dd-docs-meta-image.png]<https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations>
> Autodiscovery with JMX<https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm#autodiscovery-annotations>
> Datadog, the leading service for cloud-scale monitoring.
> docs.datadoghq.com
>
>
> This is the Environment I am using in my case.
> I am trying to integrate JMX in one my Deployment, in AKS environment and I am using Helm to upgrade my environment.
> as suggested in the documentation., I am adding the annotations in my deployment file in helm structure.
Okay, I know absolutely nothing about Azure, Kubernetes, or Helm. But it
looks like maybe this is more basic than that...
> My agent is JMX enabled, and i can see in deployment logs.
> I have configured a service for this deployment and its reflecting.
>
>
> Issue I am facing :
> I am trying to add my annotations, in the deployment file, after deployment, it should be able make the remote connection , but its showing these logs in the agent.
I will be of no help in figuring out what "annotations" you need in your
"deployment file" because - again - I know nothing about Azure, Kube,
and Helm.
> while i check the Java process., even its not reflecting this JMX process , it seems JMX process is not at all created, in my case.,
>
> error i am facing after deployment"
> ========
> JMXFetch
> ========
>
> Information
> ==================
> runtime_version : 11.0.18
> version : 0.47.8
> Initialized checks
> ==================
> tomcat
> instance_name : tomcat-10.75.12.143-1099
> message : Unable to instantiate or initialize instance 10.75.12.143:1099. Is the target JMX Server or JVM running? Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.143; nested exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
Well THAT seems straightforward.
First: is that the right IP address?
Second: on that server (10.75.12.143), is port 1099 bound-to by the
Tomcat process? Can you, for example, nc to that TCP/IP port? Have you
checked the Tomcat logs to see if there any errors during startup... for
example, any JMX-related errors?
Third: on that IP, is port 1099 reachable from the Helm (or whatever)
server that is trying to contact it? Try using "nc 10.75.12.143 1099"
and see if you can connect from Helm (or whatever).
> - name: JAVA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.port=1099
> -Dcom.sun.management.jmxremote.rmi.port=1099
> -Djava.rmi.server.hostname=$(POD_IP)
> -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.host=$(POD_IP)
Can you login to 10.75.12.143 and check to see if the JVM launch command
has (a) all of the above -D arguments in it and (b) if the host/hostname
have the expected values in them?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org