You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "exceptionfactory (via GitHub)" <gi...@apache.org> on 2025/06/11 21:33:21 UTC
[PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
exceptionfactory opened a new pull request, #10011:
URL: https://github.com/apache/nifi/pull/10011
# Summary
[NIFI-14620](https://issues.apache.org/jira/browse/NIFI-14620) Corrects the behavior of the `toDate()` Expression Language function when parsing values using the optional second argument containing a time zone identifier.
Prior to changes in [NIFI-8161](https://issues.apache.org/jira/browse/NIFI-8161), the `toDate` function used the [SimpleDateFormat.setTimeZone()](https://docs.oracle.com/javase/8/docs/api/java/text/DateFormat.html#setTimeZone-java.util.TimeZone-) method to set the Time Zone for string parsing.
With the migration to [DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) in NiFi 2.0.0-M1, the [parseBest()](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#parseBest-java.lang.CharSequence-java.time.temporal.TemporalQuery...-) method covered various types of results, but effectively ignored the Time Zone applied to the `DateTimeFormatter` using the `withTimeZone()` method.
Changes in this pull request update the `FormatUtils.parseInstant()` method, used only for Expression Language functions, to use the [DateTimeFormatter.parse()](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#parse-java.lang.CharSequence-) method. The utility method then queries the parsed `TemporalAccessor` for available fields, first constructing a `LocalDateTime` and then accounting for both the `ZoneId` passed as an argument and the `ZoneOffset` when present in the input string.
The updated approach passes existing tests for `toDate()` with and without time zone, and also passes a new unit test the compares a `java.util.Date` result with an expected instance constructed to match.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `mvn clean install -P contrib-check`
- [X] JDK 21
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2981073123
> @exceptionfactory I do not mean to belabor this but I did not see the answer to my question earlier why the date and times are skewed for the other time zones. I understand why the UTC is present but shouldn't each of the time zones have the correct date and time per their timezone? Per you explanation I would still expect to see the following
> Time Zone toDate result Expected Result
> Europe/Berlin Sat May 31 22:00:00 UTC 2025 Sun Jun 01 02:00:00 UTC 2025
> UTC Sun Jun 01 00:00:00 UTC 2025 Sun Jun 01 00:00:00 UTC 2025
> Europe/Kiev Sat May 31 21:00:00 UTC 2025 Sun Jun 01 03:00:00 UTC 2025
> America/Detroit Sun Jun 01 04:00:00 UTC 2025 Sat May 31 20:00:00 UTC 2025
> @exceptionfactory I do not mean to belabor this but I did not see the answer to my question earlier why the date and times are skewed for the other time zones. I understand why the UTC is present but shouldn't each of the time zones have the correct date and time per their timezone? Per you explanation I would still expect to see the following
> Time Zone toDate result Expected Result
> Europe/Berlin Sat May 31 22:00:00 UTC 2025 Sun Jun 01 02:00:00 UTC 2025
> UTC Sun Jun 01 00:00:00 UTC 2025 Sun Jun 01 00:00:00 UTC 2025
> Europe/Kiev Sat May 31 21:00:00 UTC 2025 Sun Jun 01 03:00:00 UTC 2025
> America/Detroit Sun Jun 01 04:00:00 UTC 2025 Sat May 31 20:00:00 UTC 2025
The apparent skew is the result of interpreting the input time zone offset relative to the system local time zone. This gets back to the behavior of `java.util.Date`.
During parsing, the `parseToInstant` method computes a `LocalDateTime`, then adjusts it based on the input Time Zone argument. After that adjustment, `Date.from()` takes the epoch time in milliseconds from the parsed `java.time.Instant`. Due to the way that `java.util.Date` always computes `toString()` relative to the system default Time Zone, the string representation is also adjusted.
This gets back to the fundamental problems with `java.util.Date` that prompted to move to more precise classes in the `java.time` package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971551019
> I assume in a different ticket, right?
Yes, a separate ticket for tracking the documentation changes, linked to this one, would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2970653222
@exceptionfactory
Thank you for the contribution. I am still noting two things which seem to be off.
I ran your fix with parameter date "20250601 and the various time zones the user had mentioned in the ticket and I get the following. Please note the time zone of my computer is UTC.
| Time Zone | toDate result|
|-----------------|--------------------|
| Europe/Berlin | Sat May 31 22:00:00 UTC 2025|
| UTC | Sun Jun 01 00:00:00 UTC 2025|
| Europe/Kiev | Sat May 31 21:00:00 UTC 2025|
| America/Detroit | Sun Jun 01 04:00:00 UTC 2025|
One thing to note is all the examples have UTC as the timezone. One of the user's issue was to have the correct timezone in the timestamp and that still is not happening. Furthermore all the later time zones than UTC such as Europe/Berlin and Europe/Kiev should have the same date Sun Jun 01 and later times 02:00:00 and 03:00:00. Also timezone America/Detroit should be an earlier date Sat May 31 and time 16:00:00.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971502013
@exceptionfactory
Thanks for responding. It seems from the user in 1.x the correct timezone was displayed when calling toDate with a timezone but now its only the local one.
Should we be changing the documentation per the user's question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971669051
@exceptionfactory
I do not mean to belabor this but I did not see the answer to my question earlier why the date and times are skewed for the other time zones. I understand why the UTC is present but shouldn't each of the time zones have the correct date and time per their timezone? Per you explanation I would still expect to see the following
| Time Zone | toDate result| Expected Result|
|-----------------|--------------------|------------------------|
| Europe/Berlin | Sat May 31 22:00:00 UTC 2025|Sun Jun 01 02:00:00 UTC 2025|
| UTC | Sun Jun 01 00:00:00 UTC 2025|Sun Jun 01 00:00:00 UTC 2025|
| Europe/Kiev | Sat May 31 21:00:00 UTC 2025|Sun Jun 01 03:00:00 UTC 2025|
| America/Detroit | Sun Jun 01 04:00:00 UTC 2025|Sat May 31 20:00:00 UTC 2025|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971468574
Thanks for following up on this issue @dan-s1.
There are several things to consider when evaluating the `java.util.Date` behavior.
The `toDate()` function returns a `java.util.Date` instance, and the `Date.toString()` representation is what is shown in the table. Internally, the class represents information using a `long` field named `fastTime` and and internal calendar with associated zone information. The zone information is what appears when calling `Date.toString()`.
The Time Zone argument to the `toDate()` function influences the behavior of the date parsing, previously `SimpleDateFormat` and now `DateTimeFormatter`. In both cases, however, the returned `java.util.Date` will display the local system time zone when using the `toString()` method.
As you described in the Jira issue, formatting `java.util.Date` can be accomplished using the `format()` EL function. The `toInstant()` function is properly more understandable in terms of actual behavior, because the resolved value is always computed according to UTC, whereas `toDate()` is more relative to local system time.
Glad to consider the issue further in Jira, but some of this is running into the issues inherent in `java.util.Date`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971532081
Yes, it is probably worth adding some clarification to the `toDate()` function, indicating that it influences parsing behavior only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #10011:
URL: https://github.com/apache/nifi/pull/10011#issuecomment-2971548978
I assume in a different ticket, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14620 Correct toDate EL Function handling of time zone [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 closed pull request #10011: NIFI-14620 Correct toDate EL Function handling of time zone
URL: https://github.com/apache/nifi/pull/10011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org