You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Chaudhary, Mohit" <mo...@dxc.com> on 2024/01/03 06:55:31 UTC
Getting False Tomcat Down Alert
Hello Team,
We have RHEL 6.10 server and configured custom script in crontab to check 8080 port is up or not, if 8080 is down then getting email alert. But some time we are facing the false alert for 2 to 3 min. When we are checking the tomcat services it was up and running fine and nothing was written in logs also.
So is it possible that 8080 port will be down for few minutes when tomcat is facing heavy traffic? Or what will be the reason for facing the false alert?.
Thanks & Regards,
Mohit Chaudhary
AMS Middleware Admin
Team DL: ams-leveraged-webadmin-offshore@dxc.com<ma...@dxc.com>
[cid:image001.png@01DA3E3E.3691CBF0]
Re: Getting Two times login page issue.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mohit,
Please don't hijack threads.
On 1/18/24 20:08, Chaudhary, Mohit wrote:
> Hello Team ,
>
> We are facing a issue with the tomcat application login URL , where it is asking twice for login to the user , with first login it will redirect again to the login page , while reentering the credentials again , it is successful ( with second attempt it is working ) and user will be redirected to the next page , this issue we have encountered after a datacenter migration for the tomcat server on Test region , the servers were cloned from the original server to the new datacenter servers, we had manually updated the new IP address on the configuration files related to new servers , but with testing part we encountering this issue ,
>
> We had 2 servers on Test region which are load balanced where 1 server is working fine ( with another server shutdown ) , no issue with login page on this server , while issue happening on another server.
>
> The application login page resides on the Lotus Domino server and authentication happens on Domino side and then it redirects the request to Apache and Tomcat servers .
>
> Please suggest on this issue .
>
> Thanks,
> Mohit
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Getting Two times login page issue.
Posted by Brian Wolfe <wo...@gmail.com>.
On Thu, Jan 18, 2024 at 8:08 PM Chaudhary, Mohit <mo...@dxc.com>
wrote:
> Hello Team ,
>
> We are facing a issue with the tomcat application login URL , where it is
> asking twice for login to the user , with first login it will redirect
> again to the login page , while reentering the credentials again , it is
> successful ( with second attempt it is working ) and user will be
> redirected to the next page , this issue we have encountered after a
> datacenter migration for the tomcat server on Test region , the servers
> were cloned from the original server to the new datacenter servers, we had
> manually updated the new IP address on the configuration files related to
> new servers , but with testing part we encountering this issue ,
>
> You haven't really indicated how this is a tomcat issue. This sounds like
it's either an application issue or a Lotus Domino SSO issue. However I
will ask some questions that may point you in the right direction. Is the
application dependent on tomcats JSession? If so , when does it determine
the userPrincipal? If not then it's an application issue. How is this SSO
setup? Header based? cookie based? agent based session? is Tomcat
responsible for connecting to the userstore to determine who the user is?
This would be done via a realm.
> We had 2 servers on Test region which are load balanced where 1 server is
> working fine ( with another server shutdown ) , no issue with login page on
> this server , while issue happening on another server.
>
So are you saying it works fine with 1 server and doesn't when 2 are
running? Are you sure its not a load balancing issue? How does your
application handle sessions? Does it replicate them? Does the application
use Affinity to keep the user going to the same server through the LB?
>
> The application login page resides on the Lotus Domino server and
> authentication happens on Domino side and then it redirects the request to
> Apache and Tomcat servers .
> Please suggest on this issue .
>
There are a lot of different things that can go wrong here and most of them
are not tomcat related.
>
> Thanks,
> Mohit
>
>
--
Thanks,
Brian Wolfe
https://www.linkedin.com/in/brian-wolfe-3136425a/
Getting Two times login page issue.
Posted by "Chaudhary, Mohit" <mo...@dxc.com>.
Hello Team ,
We are facing a issue with the tomcat application login URL , where it is asking twice for login to the user , with first login it will redirect again to the login page , while reentering the credentials again , it is successful ( with second attempt it is working ) and user will be redirected to the next page , this issue we have encountered after a datacenter migration for the tomcat server on Test region , the servers were cloned from the original server to the new datacenter servers, we had manually updated the new IP address on the configuration files related to new servers , but with testing part we encountering this issue ,
We had 2 servers on Test region which are load balanced where 1 server is working fine ( with another server shutdown ) , no issue with login page on this server , while issue happening on another server.
The application login page resides on the Lotus Domino server and authentication happens on Domino side and then it redirects the request to Apache and Tomcat servers .
Please suggest on this issue .
Thanks,
Mohit
Re: Getting False Tomcat Down Alert
Posted by Olaf Kock <to...@olafkock.de>.
On 03.01.24 10:18, Olaf Kock wrote:
> Here's an option:
>
> On 03.01.24 09:41, Chaudhary, Mohit wrote:
>> Hi,
>>
>> Please find below script code which has been written.
>>
>> STAT=`netstat -luptn | grep 8080 | awk '{print $6}'`
>> if [[ "$STAT" != "LISTEN" ]];
>> then
>> echo "Tomcat instance down" >> $MESSAGE
>> mail -s "Tomcat Instance Down on $HOSTNAME" $mailto < $MESSAGE
>>
>> Thanks & Regards,
>> Mohit Chaudhary
>
> netstat -luptn contains the "p" option (which lists the PID/Program
> name). It also contains -u, which includes UDP ports. Both most likely
> not helpful in your case, and a sign that this script was quickly
> gobbled together and not well designed.
>
> grep 8080 does not just match for a port, but will also trigger for
> any listed PID (or other output) containing those 4 characters. As
> you'll also see numeric IPV6 addresses (local or foreign ones) - and
> even local IPV6 addresses can change over time - there's another
> possibility for unintended matches.
>
> So, assuming that there is some other output from netstat that
> somewhere contains "8080", but not "LISTEN" (or maybe if the output is
> multi-line), you'll get a false positive hit.
>
> To validate that you're running into such an issue, you can add the
> grepped netstat output to the mail (before applying awk) - so either
> cache that output, or simply execute it again, piping it to $MESSAGE.
>
One more nail in the coffin:
Validate by just executing "netstat -luptn | grep 8" on the command line
(to simulate multiple hits), and look at the output that you (would) get
from awk: The -u option doesn't even result in netstat to print LISTEN
in column 6, so awk prints the PID column. And indeed, if multiple lines
match your grep, you're comparing (e.g.) "LISTEN LISTEN 34523/java" to
"LISTEN", which obviously does not match.
Which means: Your script is definitely wrong. Tomcat is not to blame.
Best,
Olaf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Getting False Tomcat Down Alert
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Olaf,
On 1/3/24 09:52, Olaf Kock wrote:
> On 03.01.24 15:34, Christopher Schultz wrote:
>> Olaf,
>>
>> +1
>>
>> The regular expression used with grep should be improved a lot.
>>
>> I would recommend at least the following:
>>
>> STAT=`netstat -luptn 2>/dev/null | grep '^tcp.*:8080[^:0-9]' | awk
>> '{print $6}'`
>
> ...or omit the UDP output by using "netstat -lptn" to begin with ;)
>
>> The improved regexp will ignore non-TCP ports and will only match on a
>> proper port-number by requiring the presence of a : and being followed
>> by anything other than a : (which would indicate it's an IPv6 address)
>> or more numbers (which could be a port number like 80800 or more of an
>> IPv6 address).
>
> wouldn't the :8080[^:0-9] still hit on an IPV6 address /ending/ in 8080?
> Still an improvement, as it'd be 1 false alarm per 4294967296 IPV6
> addresses ;)
Unlikely, as a LISTENing socket must be listening on a port number. The
port number is always at the end, so you wouldn't see a bare IPv6
address ending in :8080 with no trailing port number. :)
>> netstat is a pretty crude tool to be used, here. Why not just connect
>> to the service on port 8080 and see if it responds? The process
>> listening on the port doesn't guarantee it's actually able to serve
>> any requests...
>
> Now I'm +1'ing, and can add:
>
> As you said this, you triggered my memory: My toolbox has this gem,
> checking for http-status 200:
>
> |#!/bin/bash status_code=$(curl --write-out %{http_code} --silent
> --output /dev/null http://localhost:8080/) if [[ "$status_code" -ne 200
> ]] ; then echo "Tomcat Status: $status_code" | mail -s "Tomcat Down?"
> "somebody@example.com" -r "STATUS_CHECKER" else exit 0 fi |
Yeah, HTTP-based status-checks are indeed pretty basic. You don't have
to over-think them ;)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Getting False Tomcat Down Alert
Posted by Olaf Kock <to...@olafkock.de>.
On 03.01.24 15:34, Christopher Schultz wrote:
> Olaf,
>
> +1
>
> The regular expression used with grep should be improved a lot.
>
> I would recommend at least the following:
>
> STAT=`netstat -luptn 2>/dev/null | grep '^tcp.*:8080[^:0-9]' | awk
> '{print $6}'`
...or omit the UDP output by using "netstat -lptn" to begin with ;)
> The improved regexp will ignore non-TCP ports and will only match on a
> proper port-number by requiring the presence of a : and being followed
> by anything other than a : (which would indicate it's an IPv6 address)
> or more numbers (which could be a port number like 80800 or more of an
> IPv6 address).
wouldn't the :8080[^:0-9] still hit on an IPV6 address /ending/ in 8080?
Still an improvement, as it'd be 1 false alarm per 4294967296 IPV6
addresses ;)
> netstat is a pretty crude tool to be used, here. Why not just connect
> to the service on port 8080 and see if it responds? The process
> listening on the port doesn't guarantee it's actually able to serve
> any requests...
Now I'm +1'ing, and can add:
As you said this, you triggered my memory: My toolbox has this gem,
checking for http-status 200:
|#!/bin/bash status_code=$(curl --write-out %{http_code} --silent
--output /dev/null http://localhost:8080/) if [[ "$status_code" -ne 200
]] ; then echo "Tomcat Status: $status_code" | mail -s "Tomcat Down?"
"somebody@example.com" -r "STATUS_CHECKER" else exit 0 fi |
Olaf
Re: Getting False Tomcat Down Alert
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Olaf,
On 1/3/24 04:18, Olaf Kock wrote:
> Here's an option:
>
> On 03.01.24 09:41, Chaudhary, Mohit wrote:
>> Hi,
>>
>> Please find below script code which has been written.
>>
>> STAT=`netstat -luptn | grep 8080 | awk '{print $6}'`
>> if [[ "$STAT" != "LISTEN" ]];
>> then
>> echo "Tomcat instance down" >> $MESSAGE
>> mail -s "Tomcat Instance Down on $HOSTNAME" $mailto < $MESSAGE
>>
>> Thanks & Regards,
>> Mohit Chaudhary
>
> netstat -luptn contains the "p" option (which lists the PID/Program
> name). It also contains -u, which includes UDP ports. Both most likely
> not helpful in your case, and a sign that this script was quickly
> gobbled together and not well designed.
>
> grep 8080 does not just match for a port, but will also trigger for any
> listed PID (or other output) containing those 4 characters. As you'll
> also see numeric IPV6 addresses (local or foreign ones) - and even local
> IPV6 addresses can change over time - there's another possibility for
> unintended matches.
>
> So, assuming that there is some other output from netstat that somewhere
> contains "8080", but not "LISTEN" (or maybe if the output is
> multi-line), you'll get a false positive hit.
>
> To validate that you're running into such an issue, you can add the
> grepped netstat output to the mail (before applying awk) - so either
> cache that output, or simply execute it again, piping it to $MESSAGE.
+1
The regular expression used with grep should be improved a lot.
I would recommend at least the following:
STAT=`netstat -luptn 2>/dev/null | grep '^tcp.*:8080[^:0-9]' | awk
'{print $6}'`
In my Linux environment, if this check isn't being run by root, it will
print this message on stderr:
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
This is why I've added the 2>/dev/null to the netstat command.
The improved regexp will ignore non-TCP ports and will only match on a
proper port-number by requiring the presence of a : and being followed
by anything other than a : (which would indicate it's an IPv6 address)
or more numbers (which could be a port number like 80800 or more of an
IPv6 address).
netstat is a pretty crude tool to be used, here. Why not just connect to
the service on port 8080 and see if it responds? The process listening
on the port doesn't guarantee it's actually able to serve any requests...
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Getting False Tomcat Down Alert
Posted by Olaf Kock <to...@olafkock.de>.
Here's an option:
On 03.01.24 09:41, Chaudhary, Mohit wrote:
> Hi,
>
> Please find below script code which has been written.
>
> STAT=`netstat -luptn | grep 8080 | awk '{print $6}'`
> if [[ "$STAT" != "LISTEN" ]];
> then
> echo "Tomcat instance down" >> $MESSAGE
> mail -s "Tomcat Instance Down on $HOSTNAME" $mailto < $MESSAGE
>
> Thanks & Regards,
> Mohit Chaudhary
netstat -luptn contains the "p" option (which lists the PID/Program
name). It also contains -u, which includes UDP ports. Both most likely
not helpful in your case, and a sign that this script was quickly
gobbled together and not well designed.
grep 8080 does not just match for a port, but will also trigger for any
listed PID (or other output) containing those 4 characters. As you'll
also see numeric IPV6 addresses (local or foreign ones) - and even local
IPV6 addresses can change over time - there's another possibility for
unintended matches.
So, assuming that there is some other output from netstat that somewhere
contains "8080", but not "LISTEN" (or maybe if the output is
multi-line), you'll get a false positive hit.
To validate that you're running into such an issue, you can add the
grepped netstat output to the mail (before applying awk) - so either
cache that output, or simply execute it again, piping it to $MESSAGE.
Olaf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Getting False Tomcat Down Alert
Posted by "Chaudhary, Mohit" <mo...@dxc.com>.
Hi,
Please find below script code which has been written.
STAT=`netstat -luptn | grep 8080 | awk '{print $6}'`
if [[ "$STAT" != "LISTEN" ]];
then
echo "Tomcat instance down" >> $MESSAGE
mail -s "Tomcat Instance Down on $HOSTNAME" $mailto < $MESSAGE
Thanks & Regards,
Mohit Chaudhary
-----Original Message-----
From: Olaf Kock <to...@olafkock.de>
Sent: Wednesday, January 3, 2024 2:03 PM
To: users@tomcat.apache.org
Subject: Re: Getting False Tomcat Down Alert
[You don't often get email from tomcat@olafkock.de. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
On 03.01.24 07:55, Chaudhary, Mohit wrote:
>
> Hello Team,
>
> We have RHEL 6.10 server and configured custom script in crontab to
> check 8080 port is up or not, if 8080 is down then getting email
> alert. But some time we are facing the false alert for 2 to 3 min.
> When we are checking the tomcat services it was up and running fine
> and nothing was written in logs also.
>
> So is it possible that 8080 port will be down for few minutes when
> tomcat is facing heavy traffic? Or what will be the reason for facing
> the false alert?.
>
To me, the question is rather: *How* does your custom script check port
8080 being up? Because you say that you're checking as well, and this result disagrees with the result of the manual check. With two different results, I'd first inspect the custom script for its behavior, and suspect it to generate a false positive.
It being a script implies that you might be able to share it (or the relevant parts of it) here (?)
Olaf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Getting False Tomcat Down Alert
Posted by Olaf Kock <to...@olafkock.de>.
On 03.01.24 07:55, Chaudhary, Mohit wrote:
>
> Hello Team,
>
> We have RHEL 6.10 server and configured custom script in crontab to
> check 8080 port is up or not, if 8080 is down then getting email
> alert. But some time we are facing the false alert for 2 to 3 min.
> When we are checking the tomcat services it was up and running fine
> and nothing was written in logs also.
>
> So is it possible that 8080 port will be down for few minutes when
> tomcat is facing heavy traffic? Or what will be the reason for facing
> the false alert?.
>
To me, the question is rather: *How* does your custom script check port
8080 being up? Because you say that you're checking as well, and this
result disagrees with the result of the manual check. With two different
results, I'd first inspect the custom script for its behavior, and
suspect it to generate a false positive.
It being a script implies that you might be able to share it (or the
relevant parts of it) here (?)
Olaf