You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by David <ra...@gmail.com> on 2022/10/27 15:31:01 UTC
Tomcat/Java app timezone radomly changes during operation.
Hi all,
I've experienced an issue since the morning of the 21st that I'm
hoping to get some direction on for where to look.
An app uses the date/time to set a timeout for a password reset.
This had been working fine for years and suddenly it failed. A restart of
tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr
and now is averaging a 10 minute or so working duration between tomcat
restarts.
Changing the logging in the app showed that the issue is due to it
sending UTC to the DB while it is broken. Restarting Tomcat results in CDT
being sent for a while until randomly it switches again.
RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
ntp is on, chrony is syncing, Java states correct time when queried
however unsure if it's JDK or JRE when targeted. OS time is good.
When I redeploy the app, log timestamps for the app are in UTC as well
until restarting tomcat. During the issue the log timestamp remains in
CDT as expected, even though values passed are UTC.
I have explicitly defined the timezone in setenv.sh with no change in
behavior.
Any thoughts as what to investigate are greatly appreciated.
Thanks,
David
RE: Tomcat/Java app timezone radomly changes during operation.
Posted by Hiran CHAUDHURI <hi...@amadeus.com.INVALID>.
CONFIDENTIAL & RESTRICTED
Does the system time (OS) change in a similar way?
Is it always the change of instead using CDT the timestamps switch to UTC?
Do the times at least match to some degree? I mean is it just the wrong timezone (and same hour of the day), or is it the same instant in time just represented from a different timezone (02:06 CDT -> 09:06 CEST)?
If the system time remains stable, it is related to the JVM. Also the fact that a restart fixes the problem is an indication the system time is correct. Remains the fact that the time is lost after decreasing intervals.
So there might be the question if the application can configure timezone/system time and to find out whether it does it. If you find out which java methods need to be called for this to change then you could check the source code for such calls, or at least set breakpoints to be informed immediately when such a line of code is reached.
Hiran
-----Original Message-----
From: David <ra...@gmail.com>
Sent: Thursday, October 27, 2022 17:31
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Tomcat/Java app timezone radomly changes during operation.
CAUTION: External mail. Be careful with links and attachments.
Hi all,
I've experienced an issue since the morning of the 21st that I'm hoping to get some direction on for where to look.
An app uses the date/time to set a timeout for a password reset.
This had been working fine for years and suddenly it failed. A restart of tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr and now is averaging a 10 minute or so working duration between tomcat restarts.
Changing the logging in the app showed that the issue is due to it sending UTC to the DB while it is broken. Restarting Tomcat results in CDT being sent for a while until randomly it switches again.
RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29 ntp is on, chrony is syncing, Java states correct time when queried however unsure if it's JDK or JRE when targeted. OS time is good.
When I redeploy the app, log timestamps for the app are in UTC as well
until restarting tomcat. During the issue the log timestamp remains in
CDT as expected, even though values passed are UTC.
I have explicitly defined the timezone in setenv.sh with no change in behavior.
Any thoughts as what to investigate are greatly appreciated.
Thanks,
David
IMPORTANT - CONFIDENTIALITY NOTICE - This e-mail is intended only for the use of the individual or entity shown above as addressees . It may contain information which is privileged, confidential or otherwise protected from disclosure under applicable laws . If the reader of this transmission is not the intended recipient, you are hereby notified that any dissemination, printing, distribution, copying, disclosure or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this transmission in error, please immediately notify us by reply e-mail or using the address below and delete the message and any attachments from your system. Amadeus Data Processing GmbH Geschaftsfuhrer: Sven Fuhrmeister Sitz der Gesellschaft: Erding HR Munchen 212770 Berghamer Strasse 6 85435 Erding Germany.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by David <ra...@gmail.com>.
Also to note, I'm unsure if the TZ in the header is being sent or if there
would be a change in those. The F5 offloads connections to the box, so
every session is a local SNAT from the F5, but it's something to
investigate. Thanks again for your leads everyone!
On Fri, Oct 28, 2022, 09:22 David <ra...@gmail.com> wrote:
> I'll see if I can answer all queries.
>
> The server doesn't migrate, at least not across timezones. It is a
> nutanix virtual though, so migration between hosts locally is possible.
>
> There is a Java app that allows a user to reset their password. The db
> is called to retrieve a generated code for them to use to reset, it also
> validates the user hasn't done this more than 3 times since having a valid
> login, and sets an expiry for the duration of this code (4hrs).
>
> There is a dbutil Java app that does use SimpleDateFormat, so I will look
> into the bug provided.
>
> When the switch happens, it will show that the date being sent to the DB
> by the Java reset pw app is sending values in UTC. It is instant and
> precisely on time, since it's +5 hours, and the pw reset is only valid for
> 4, it breaks this functionality. The Java app lives in tomcat and is a
> subset of the main website app.
>
> We have lower environments where this doesn't occur. The app code is the
> same everywhere, Java and OS times are stable everywhere as well.
>
> I did notice that stuck valve detection had a lot of hits during one
> occurrence, tomcat slowed as connection threads climbed, and the switch to
> utc being sent to the db happened at this time. I was thinking it may be a
> db2 contention issue based on this, normally that would cause connections
> to back up and tomcat to run out of threads though, I've never seen it
> change cdt to udt. I'll dig more into the SimpleDateFormat, as it may be
> the cause but is triggered by load/db contention. Lower environments that
> don't show the issue don't have much traffic as they are for dev/uat/etc.
>
> Thanks all!
> David
>
> On Fri, Oct 28, 2022, 02:51 Konstantin Kolinko <kn...@gmail.com>
> wrote:
>
>> чт, 27 окт. 2022 г. в 18:31, David <ra...@gmail.com>:
>> >
>> > Hi all,
>> >
>> > I've experienced an issue since the morning of the 21st that I'm
>> > hoping to get some direction on for where to look.
>> >
>> > An app uses the date/time to set a timeout for a password reset.
>> > This had been working fine for years and suddenly it failed. A restart
>> of
>> > tomcat allowed it to work for a day, then 12 hours, then 5 hours, then
>> 1 hr
>> > and now is averaging a 10 minute or so working duration between tomcat
>> > restarts.
>> >
>> > Changing the logging in the app showed that the issue is due to it
>> > sending UTC to the DB while it is broken. Restarting Tomcat results in
>> CDT
>> > being sent for a while until randomly it switches again.
>> >
>> > RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
>>
>> Looking at the changelog since 9.0.29 onwards,
>> there was the following issue, fixed in 9.0.34 two years ago:
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64226
>> "Tomcat 9 can return HTTP date headers in timezone other than GMT"
>>
>> Fixed by commit
>>
>> https://github.com/apache/tomcat/commit/f64a2a7150bff01bca479c7c319b7e8db879df26
>>
>> It is about how parsing a date time string may affect formatting time
>> if the same SimpleDateFormat instance is reused. It is triggered by a
>> client sending a header using a different time zone.
>>
>>
>> It is unlikely your issue with the database (unless your application
>> uses Tomcat internal classes such as ConcurrentDateFormat or
>> FastHttpDateFormat), but it affects HTTP headers generated by Tomcat.
>>
>> Best regards,
>> Konstantin Kolinko
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by George Sexton <ge...@mhsoftware.com.INVALID>.
When you say the time is wrong, how do you know this. Are you using a
DateFormatter?
Are you instantiating one, and re-using it? Could some code re-using a
DateFormatter be changing the time zone?
On 10/28/2022 8:22 AM, David wrote:
> I'll see if I can answer all queries.
>
> The server doesn't migrate, at least not across timezones. It is a nutanix
> virtual though, so migration between hosts locally is possible.
>
> There is a Java app that allows a user to reset their password. The db is
> called to retrieve a generated code for them to use to reset, it also
> validates the user hasn't done this more than 3 times since having a valid
> login, and sets an expiry for the duration of this code (4hrs).
>
> There is a dbutil Java app that does use SimpleDateFormat, so I will look
> into the bug provided.
>
> When the switch happens, it will show that the date being sent to the DB by
> the Java reset pw app is sending values in UTC. It is instant and
> precisely on time, since it's +5 hours, and the pw reset is only valid for
> 4, it breaks this functionality. The Java app lives in tomcat and is a
> subset of the main website app.
>
> We have lower environments where this doesn't occur. The app code is the
> same everywhere, Java and OS times are stable everywhere as well.
>
> I did notice that stuck valve detection had a lot of hits during one
> occurrence, tomcat slowed as connection threads climbed, and the switch to
> utc being sent to the db happened at this time. I was thinking it may be a
> db2 contention issue based on this, normally that would cause connections
> to back up and tomcat to run out of threads though, I've never seen it
> change cdt to udt. I'll dig more into the SimpleDateFormat, as it may be
> the cause but is triggered by load/db contention. Lower environments that
> don't show the issue don't have much traffic as they are for dev/uat/etc.
>
> Thanks all!
> David
>
> On Fri, Oct 28, 2022, 02:51 Konstantin Kolinko <kn...@gmail.com>
> wrote:
>
>> чт, 27 окт. 2022 г. в 18:31, David <ra...@gmail.com>:
>>> Hi all,
>>>
>>> I've experienced an issue since the morning of the 21st that I'm
>>> hoping to get some direction on for where to look.
>>>
>>> An app uses the date/time to set a timeout for a password reset.
>>> This had been working fine for years and suddenly it failed. A restart
>> of
>>> tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1
>> hr
>>> and now is averaging a 10 minute or so working duration between tomcat
>>> restarts.
>>>
>>> Changing the logging in the app showed that the issue is due to it
>>> sending UTC to the DB while it is broken. Restarting Tomcat results in
>> CDT
>>> being sent for a while until randomly it switches again.
>>>
>>> RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
>> Looking at the changelog since 9.0.29 onwards,
>> there was the following issue, fixed in 9.0.34 two years ago:
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64226
>> "Tomcat 9 can return HTTP date headers in timezone other than GMT"
>>
>> Fixed by commit
>>
>> https://github.com/apache/tomcat/commit/f64a2a7150bff01bca479c7c319b7e8db879df26
>>
>> It is about how parsing a date time string may affect formatting time
>> if the same SimpleDateFormat instance is reused. It is triggered by a
>> client sending a header using a different time zone.
>>
>>
>> It is unlikely your issue with the database (unless your application
>> uses Tomcat internal classes such as ConcurrentDateFormat or
>> FastHttpDateFormat), but it affects HTTP headers generated by Tomcat.
>>
>> Best regards,
>> Konstantin Kolinko
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
--
George Sexton
(303) 438 9585 x102
MH Software, Inc.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by David <ra...@gmail.com>.
I'll see if I can answer all queries.
The server doesn't migrate, at least not across timezones. It is a nutanix
virtual though, so migration between hosts locally is possible.
There is a Java app that allows a user to reset their password. The db is
called to retrieve a generated code for them to use to reset, it also
validates the user hasn't done this more than 3 times since having a valid
login, and sets an expiry for the duration of this code (4hrs).
There is a dbutil Java app that does use SimpleDateFormat, so I will look
into the bug provided.
When the switch happens, it will show that the date being sent to the DB by
the Java reset pw app is sending values in UTC. It is instant and
precisely on time, since it's +5 hours, and the pw reset is only valid for
4, it breaks this functionality. The Java app lives in tomcat and is a
subset of the main website app.
We have lower environments where this doesn't occur. The app code is the
same everywhere, Java and OS times are stable everywhere as well.
I did notice that stuck valve detection had a lot of hits during one
occurrence, tomcat slowed as connection threads climbed, and the switch to
utc being sent to the db happened at this time. I was thinking it may be a
db2 contention issue based on this, normally that would cause connections
to back up and tomcat to run out of threads though, I've never seen it
change cdt to udt. I'll dig more into the SimpleDateFormat, as it may be
the cause but is triggered by load/db contention. Lower environments that
don't show the issue don't have much traffic as they are for dev/uat/etc.
Thanks all!
David
On Fri, Oct 28, 2022, 02:51 Konstantin Kolinko <kn...@gmail.com>
wrote:
> чт, 27 окт. 2022 г. в 18:31, David <ra...@gmail.com>:
> >
> > Hi all,
> >
> > I've experienced an issue since the morning of the 21st that I'm
> > hoping to get some direction on for where to look.
> >
> > An app uses the date/time to set a timeout for a password reset.
> > This had been working fine for years and suddenly it failed. A restart
> of
> > tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1
> hr
> > and now is averaging a 10 minute or so working duration between tomcat
> > restarts.
> >
> > Changing the logging in the app showed that the issue is due to it
> > sending UTC to the DB while it is broken. Restarting Tomcat results in
> CDT
> > being sent for a while until randomly it switches again.
> >
> > RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
>
> Looking at the changelog since 9.0.29 onwards,
> there was the following issue, fixed in 9.0.34 two years ago:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64226
> "Tomcat 9 can return HTTP date headers in timezone other than GMT"
>
> Fixed by commit
>
> https://github.com/apache/tomcat/commit/f64a2a7150bff01bca479c7c319b7e8db879df26
>
> It is about how parsing a date time string may affect formatting time
> if the same SimpleDateFormat instance is reused. It is triggered by a
> client sending a header using a different time zone.
>
>
> It is unlikely your issue with the database (unless your application
> uses Tomcat internal classes such as ConcurrentDateFormat or
> FastHttpDateFormat), but it affects HTTP headers generated by Tomcat.
>
> Best regards,
> Konstantin Kolinko
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by Konstantin Kolinko <kn...@gmail.com>.
чт, 27 окт. 2022 г. в 18:31, David <ra...@gmail.com>:
>
> Hi all,
>
> I've experienced an issue since the morning of the 21st that I'm
> hoping to get some direction on for where to look.
>
> An app uses the date/time to set a timeout for a password reset.
> This had been working fine for years and suddenly it failed. A restart of
> tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr
> and now is averaging a 10 minute or so working duration between tomcat
> restarts.
>
> Changing the logging in the app showed that the issue is due to it
> sending UTC to the DB while it is broken. Restarting Tomcat results in CDT
> being sent for a while until randomly it switches again.
>
> RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
Looking at the changelog since 9.0.29 onwards,
there was the following issue, fixed in 9.0.34 two years ago:
https://bz.apache.org/bugzilla/show_bug.cgi?id=64226
"Tomcat 9 can return HTTP date headers in timezone other than GMT"
Fixed by commit
https://github.com/apache/tomcat/commit/f64a2a7150bff01bca479c7c319b7e8db879df26
It is about how parsing a date time string may affect formatting time
if the same SimpleDateFormat instance is reused. It is triggered by a
client sending a header using a different time zone.
It is unlikely your issue with the database (unless your application
uses Tomcat internal classes such as ConcurrentDateFormat or
FastHttpDateFormat), but it affects HTTP headers generated by Tomcat.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by Mark Thomas <ma...@apache.org>.
David,
I strongly suspect TimeZone.setDefault() is being called somewhere. I
can confirm it isn't Tomcat calling it. If the problem was preceded by
any application updates, I'd start looking there.
Mark
On 27/10/2022 16:31, David wrote:
> Hi all,
>
> I've experienced an issue since the morning of the 21st that I'm
> hoping to get some direction on for where to look.
>
> An app uses the date/time to set a timeout for a password reset.
> This had been working fine for years and suddenly it failed. A restart of
> tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr
> and now is averaging a 10 minute or so working duration between tomcat
> restarts.
>
> Changing the logging in the app showed that the issue is due to it
> sending UTC to the DB while it is broken. Restarting Tomcat results in CDT
> being sent for a while until randomly it switches again.
>
> RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
> ntp is on, chrony is syncing, Java states correct time when queried
> however unsure if it's JDK or JRE when targeted. OS time is good.
>
> When I redeploy the app, log timestamps for the app are in UTC as well
> until restarting tomcat. During the issue the log timestamp remains in
> CDT as expected, even though values passed are UTC.
>
> I have explicitly defined the timezone in setenv.sh with no change in
> behavior.
>
> Any thoughts as what to investigate are greatly appreciated.
>
> Thanks,
> David
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/Java app timezone radomly changes during operation.
Posted by Chuck Caldarale <n8...@gmail.com>.
> On Oct 28, 2022, at 02:10, Terence M. Bandoian <te...@tmbsw.com> wrote:
>
> Have you tried setting the JVM time zone with an environment variable or JVM argument or with TimeZone.setDefault? I think Mark Thomas mentioned earlier that Tomcat may invoke TimeZone.setDefault.
No, Mark said the opposite. Tomcat itself does not invoke TimeZone.setDefault(), but if an application running within Tomcat does, it affects the entire JVM.
- Chuck
Re: Aw: Tomcat/Java app timezone radomly changes during operation.
Posted by "Terence M. Bandoian" <te...@tmbsw.com>.
On 10/27/2022 6:27 PM, Peter Rader wrote:
> Hi David,
>
> is it a moving server? We had similar issues on a airborn server crossing nation-borders rapidly.
>
> 10 minutes is unusual. The lowest timezone-change is 15 minutes afaik.
>
> Kind regards
>
>> Hi all,
>>
>> I've experienced an issue since the morning of the 21st that I'm
>> hoping to get some direction on for where to look.
>>
>> An app uses the date/time to set a timeout for a password reset.
>> This had been working fine for years and suddenly it failed. A restart of
>> tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr
>> and now is averaging a 10 minute or so working duration between tomcat
>> restarts.
>>
>> Changing the logging in the app showed that the issue is due to it
>> sending UTC to the DB while it is broken. Restarting Tomcat results in CDT
>> being sent for a while until randomly it switches again.
>>
>> RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
>> ntp is on, chrony is syncing, Java states correct time when queried
>> however unsure if it's JDK or JRE when targeted. OS time is good.
>>
>> When I redeploy the app, log timestamps for the app are in UTC as well
>> until restarting tomcat. During the issue the log timestamp remains in
>> CDT as expected, even though values passed are UTC.
>>
>> I have explicitly defined the timezone in setenv.sh with no change in
>> behavior.
>>
>> Any thoughts as what to investigate are greatly appreciated.
>>
>> Thanks,
>> David
Have you tried setting the JVM time zone with an environment variable or JVM argument or with TimeZone.setDefault? I think Mark Thomas mentioned earlier that Tomcat may invoke TimeZone.setDefault.
What do mean when you say "sending UTC to the DB while it is broken"? Are you populating a date/time, time or timestamp column? Sending a date or time value for some other purpose? What is "sending UTC to the DB"?
Also, what do you mean by "broken"? While what is broken?
It isn't clear to me what's happening. Is the O/S time zone getting changed? Does your app run outside of Tomcat? Is your app communicating directly with the database? Is this a Java app?
-Terence Bandoian
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Aw: Tomcat/Java app timezone radomly changes during operation.
Posted by Peter Rader <p....@gmx.net>.
Hi David,
is it a moving server? We had similar issues on a airborn server crossing nation-borders rapidly.
10 minutes is unusual. The lowest timezone-change is 15 minutes afaik.
Kind regards
>
> Hi all,
>
> I've experienced an issue since the morning of the 21st that I'm
> hoping to get some direction on for where to look.
>
> An app uses the date/time to set a timeout for a password reset.
> This had been working fine for years and suddenly it failed. A restart of
> tomcat allowed it to work for a day, then 12 hours, then 5 hours, then 1 hr
> and now is averaging a 10 minute or so working duration between tomcat
> restarts.
>
> Changing the logging in the app showed that the issue is due to it
> sending UTC to the DB while it is broken. Restarting Tomcat results in CDT
> being sent for a while until randomly it switches again.
>
> RHEL 7.9, jvm 1.8.0_231-b11, Tomcat 9.0.29
> ntp is on, chrony is syncing, Java states correct time when queried
> however unsure if it's JDK or JRE when targeted. OS time is good.
>
> When I redeploy the app, log timestamps for the app are in UTC as well
> until restarting tomcat. During the issue the log timestamp remains in
> CDT as expected, even though values passed are UTC.
>
> I have explicitly defined the timezone in setenv.sh with no change in
> behavior.
>
> Any thoughts as what to investigate are greatly appreciated.
>
> Thanks,
> David
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org