You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by lo...@kreuser.name on 2023/12/08 10:26:25 UTC
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
> Am 08.12.2023 um 06:43 schrieb Bodavula, Sai Vamsi Mohan Krishna (TR Technology) <Sa...@thomsonreuters.com.INVALID>:
>
> Hey Christopher.,
> Greetings of the day.
>
>
> 1.
> Might I have confused you with posting the arguments directly ., Yeah as i just shared you the annotations with comments , to state you the stuff i am using., But in my deployment ., I am using them in catalina opts., and trying to call them from values.yaml., which looks like this :
>
> javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local "
>
> and I am referring the word "javaVMMemoryArgument" from values yaml and calling it in Catalina_opts, so that it would fetch all these annotations as mentioned above, during the deployment. This is my deployment part., where I am referring to the above values from values.yaml
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
I compare with my CATALINA_OPTS (and I noticed that some of my options were still in JAVA_OPTS boooh!) :
export CATALINA_OPTS="${CATALINA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${HOSTNAME} -Dcom.sun.management.jmxremote.local.only=false"
I see all my options in
root@tomcat:/usr/local/tomcat# ps -aef | less
UID PID PPID C STIME TTY TIME CMD
tomcat 1 0 71 10:42 ? 00:04:02 /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat.base/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -agentlib:jdwp=transport=dt_socket,address=0.0.0.0:8000,server=y,suspend=n -XX:NativeMemoryTracking=summary -Dhostname=<hostname> -Djava.awt.headless=true -Djavax.net.ssl.trustStore=/opt/apache-tomcat.base/conf/ssl/cacerts.jks -Xlog:gc:/opt/apache-tomcat.base/logs/gc.log -Djava.security.egd=file:/dev/urandom -Dsun.net.inetaddr.ttl=60 -Djava.library.path=/usr/local/tomcat/native-jni-lib -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdk.tls.server.enableStatusRequestExtension=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<hostname> -Dcom.sun.management.jmxremote.local.only=false -javaagent:/opt/apache-tomcat.base/bin/jmx_prometheus_javaagent-0.12.0.jar=8080:/opt/apache-tomcat.base/bin/tomcat.yaml -Djava.net.debug=ssl -XX:+UnlockDiagnosticVMOptions -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/apache-tomcat.base -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/opt/apache-tomcat.base/temp org.apache.catalina.startup.Bootstrap start
Ports are opened
root@tomcat:/usr/local/tomcat# netstat -tulpn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
...
tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN -
...
tcp 2 0 0.0.0.0:8443 0.0.0.0:* LISTEN -
...
jconsole <hostname>:10001
> JMX connect success
>
>
>
> 1.
> Coming to Process., I have searched for Java process that listens on my mentioned port ie., 49151, but none of the process is listening to that process.
>
> I even tried with
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep LISTEN
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | more
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN 1/java
> tcp6 0 0 :::9109 :::* LISTEN 1/java
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':443'netstat -tulpn | grep ':443'^C
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | grep ':49151'
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
>
> which confirms me that , any of the process is being listening on the port 49151.
>
You should at least see some error when starting Tomcat. Anything in the logs or on the console?
Peter
> 2.
> I would like to request you to suggest me with a better approach ., where i am missing anything in this process!
>
>
>
>
>
>
> Thanks & Regards,
> --
>
> SAI VAMSI .B
> Senior DevOps Engineer
>
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Thursday, December 7, 2023 23:33
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Hello Christopher.
>> Thanks for the response!
>>
>> Let me walk you through the Environment.,
>> I have a cluster and I am installing multiple microservices in the cluster. And one of the microservices I am integrating the JMX on of the microservices.,
>> and the mentioned IP of the pod is being changed dynamically after each deployment .
>>
>> Here are the annotations that I am using in my deployment:
>> - name: JAVA_OPTS
>> # value: >-
>> # -Dcom.sun.management.jmxremote
>> # -Dcom.sun.management.jmxremote.authenticate=false
>> # -Dcom.sun.management.jmxremote.ssl=false
>> # -Dcom.sun.management.jmxremote.local.only=false
>> # -Dcom.sun.management.jmxremote.port=49151
>> # -Dcom.sun.management.jmxremote.rmi.port=49151
>> # -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
>
> If that's YAML, then all of the above is comments. Those lines do nothing.
>
>> as i said , I have implemented these in three scenarios., Catalina_opts / Java_opts / Java_Tool_opts
>
> You should definitely use CATALINA_OPTS and none of the other things you
> mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
> don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
> Tomcat-related scripts, and you'll find that even "shutdown" causes an
> error... because it's trying to initialize JMX when shutting-down, which
> is again useless.
>
>> I have tried deploying catalina_ots and Java_opts as in the case of above mentioned.
>> I also tried other way, mentioning all these in the values.yaml file ( I am using Helm Deployment to upgrade my deployments) and referring these annotations as , and mentioning them in the Value
>> env:
>> - name: CATALINA_OPTS
>> value: {{ .Values.deployment.javaVMMemoryArgument }}
>>
>>
>> getting netstat output from my pod:
>> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>> tcp6 0 0 :::10109 :::* LISTEN 1/java
>> tcp6 0 0 :::9109 :::* LISTEN 1/java
>>
>> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep tomcat
>> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep --color=auto tomcat
>
> Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
> 'tomcat' does not appear in a normal JVM launch command for Tomcat.
>
>> right now , I am passing my args with Catalina_opts in my deployment
>> and I am not able to get any tomcat process pid while i check with
>> netstat -apn , I can see multiple Java Process but not single tomcat process
>>
>> My Jmx port is being configured as a service., so that it could be
>> bounded with Tcp protocol externally to the container., and allow all
>> tcp connections.
>>
>> Hope I am able to provide you some detailed information., Please let
>> me know., if I can help you with anything in further!
>
> I think maybe you are both configuring your YAML file incorrectly and
> also looking for the process incorrectly.
>
> -chris
>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Wednesday, December 6, 2023 20:40
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsi,
>>
>> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> I have added my annotations in Catalina_opts ., and I am not able to see any Java Process with the JMX, neither the port is being listened.
>>> It came back to previous stage where Annotations are not on the JVM .,
>>> Came back to previous stage, where connection refused error., as follows.
>>>
>>> Could not initialize instance: tomcat-10.75.12.136-49151:
>>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested exception is:
>>> java.net.ConnectException: Connection refused (Connection refused)]
>>> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
>>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
>>> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
>>> at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
>>> at org.datadog.jmxfetch.App.init(App.java:917)
>>> at org.datadog.jmxfetch.App.start(App.java:413)
>>> at org.datadog.jmxfetch.App.run(App.java:173)
>>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
>>> at org.datadog.jmxfetch.App.main(App.java:91)
>>
>> Right: the DataDog server is unable to connect to your appication via
>> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
>> and that DataDog isn't running on that server. It's on some other IP, right?
>>
>>> Here we were not able to find any process , to confirm that atleast
>>> port is being listened to.
>> What is your exact CATALINA_OPTS value (masking any secrets that may be
>> in there).
>>
>> Is Tomcat definitely running? Can you prove that with a "ps" listing,
>> for example? That will also confirm it's running with the proper
>> CATALINA_OPTS settings.
>>
>> Can you run "netstat" and show all of the ports your Tomcat process is
>> listening to?
>>
>> I'm assuming that your Tomcat process is being hosted in some kind of
>> container. Is the IP address 10.75.12.136 the IP of the container, or
>> the IP of the host which hosts the container? Is your JMX port
>> configured to be visible external to the container itself, or only
>> internally to the container?
>>
>> -chris
>>
>>
>>
>>> Thanks & Regards,
>>> --
>>>
>>> SAI VAMSI .B
>>> Senior DevOps Engineer
>>>
>>> ________________________________
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Sent: Tuesday, December 5, 2023 19:19
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>
>>> Sai Vamsa,
>>>
>>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>> Thanks for the response..
>>>>
>>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of Java_Opts and Java_tool_options.,
>>>>
>>>> But its not even working out., Java process is not getting created., and port is not bound to any process.
>>>> Let me know if I am missing anything.
>>>
>>> Okay. Can you please summarize the current situation? I think you've
>>> made a few changes, etc. so it would help to know the current state.
>>>
>>> Stop Tomcat and any other related services, delete all appropriate log
>>> files, etc. and start Tomcat (or attempt to). Then post any output that
>>> is relevant, and if you get things like "address in use" then please
>>> post the details about which process id owns that port and the full
>>> command-line that launched that process.
>>>
>>> Thanks,
>>> -chris
>>>
>>>> ________________________________
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Sent: Monday, December 4, 2023 23:22
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>>
>>>> Sai Vamsi,
>>>>
>>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>>> Firstly thanks for adding a point me in asking me to check, if the annotations are reflecting in the Java process, which opened me a door to add the concerned annotations in correct place., by adding in java_tool_options in stead of Java_opts.
>>>>
>>>> You will probably want to use CATALINA_OPTS instead of any of the other
>>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>>>> need to have the JMX subsystem starting-up and trying to grab a port.
>>>>
>>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>>>> processes, like jps for example...
>>>>
>>>>> yeah they are reflecting and creating a Java Process.
>>>>> but I am facing a problem here., while i am checking JSP, Thats : the port i am using here to enable JMX is been opening a process with the mentioned port and at the same time shows port is being used.:
>>>>>
>>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local -javaagent:/datadog-lib/dd-java-agent.jar -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 49151; nested exception is:
>>>>> java.net.BindException: Address already in use (Bind failed)
>>>>
>>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>>>> to your port which is aready bound by your Tomcat process.
>>>>
>>>> The solution is to use only CATALINA_OPTS to set these options.
>>>>
>>>> -chris
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>