You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/01/25 21:45:02 UTC
[GitHub] [nifi] turcsanyip opened a new pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
turcsanyip opened a new pull request #4781:
URL: https://github.com/apache/nifi/pull/4781
…ed forms before/after database operations
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-YYYY._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564566112
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
I could not go on that way because `testConvertToAvroStreamForDateTime()` is used in two places and the other really expects a Date object.
However, I found a simpler way to convert the Date to the expected number of days, and then to assert the raw value from Avro against that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564138178
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
Indeed... what do you think about this?
```
public static Date convertDateToUTC(Date dateLocalTZ) {
ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault());
ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC"));
Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli());
return dateUTC;
}
```
No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564073721
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
Can the `ZoneId.of("UTC")` reference can be replaced with `ZoneOffset.UTC`?
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
##########
@@ -26,12 +27,12 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Array;
+import java.sql.Date;
Review comment:
Changing `java.util.Date` to `java.sql.Date` changes the value of `DATE_CLASS_NAME`, does that have any other implications that should be considered?
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
The method naming and implementation are a little confusing on initial read. The `atZone()` method returns a `ZonedDateTime` and `toLocalDate()` return the `java.time.LocalDate` portion, but does that actually convert the date to the system default time zone? It may be helpful to break this into multiple lines as it is a bit hard to follow.
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
Instead of performing multiple conversions for the test, could this comparison be simplified by comparing `daysSinceEpoch` to the actual number of expected days? That would make the assert much clearer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564092023
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
##########
@@ -26,12 +27,12 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Array;
+import java.sql.Date;
Review comment:
It was intentional because it seemed to be a bug. `ResultSet.getMetaData().getColumnClassName()` would never return `java.util.Date` but the SQL date/time types.
However, now I see that this code section is related to QueryRecord and it may use `java.util.Date`. I'll revert it back.
Thanks for the heads up that made me double check it in more detail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4781:
URL: https://github.com/apache/nifi/pull/4781
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip edited a comment on pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip edited a comment on pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#issuecomment-767588990
@exceptionfactory Thanks for the review, implemented your suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564138178
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
Indeed... what do you think about this?
```
public static Date convertDateToUTC(Date dateLocalTZ) {
ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault());
ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC"));
Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli());
return dateUTC;
}
```
No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here.
The same can be applied to `convertDateToLocalTZ()` similarly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564103224
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the inputs of the test cases).
I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method.
As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think?
I mean:
```
int expectedDaysSinceEpoch = 17444;
assertDate.accept(record, expectedDaysSinceEpoch);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564092023
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
##########
@@ -26,12 +27,12 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Array;
+import java.sql.Date;
Review comment:
It was intentional because it seemed to be a bug. `ResultSet.getMetaData().getColumnClassName()` would never return `java.util.Date` but the SQL date/time types.
However, now I see that this code section is related to QueryRecord and it may use `java.util.Date`. I'll revert it back.
Thanks for the heads up that made me double check it in more detail.
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
`atStartOfDay()` has `ZoneId` parameter only.
In my understanding `ZoneOffset` is DST-agnostic, so the `ZoneId` is the right choice in any case.
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the input of the test case).
I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method.
As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think?
I mean
```
int expectedDaysSinceEpoch = 17444
assertDate.accept(record, expectedDaysSinceEpoch);
```
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the inputs of the test cases).
I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method.
As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think?
I mean:
```
int expectedDaysSinceEpoch = 17444;
assertDate.accept(record, expectedDaysSinceEpoch);
```
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
Indeed... what do you think about this?
```
public static Date convertDateToUTC(Date dateLocalTZ) {
ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault());
ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC"));
Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli());
return dateUTC;
}
```
No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here.
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
Indeed... what do you think about this?
```
public static Date convertDateToUTC(Date dateLocalTZ) {
ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault());
ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC"));
Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli());
return dateUTC;
}
```
No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here.
The same can be applied to `convertDateToLocalTZ()` similarly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564218598
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
Reading the documentation, `ZoneOffset` extends `ZoneId` so would `ZoneOffset.UTC` function the same as `ZoneId.of("UTC")`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#issuecomment-767784466
@turcsanyip Thanks for adding the tests, I confirmed the changes function as designed and cover the lines changed in `PutDatabaseRecord` and `ResultSetRecordSet`. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#issuecomment-767646267
@turcsanyip Thanks for making the changes. I confirmed that several of the tests run in different time zones as designed. I noticed that the changes for `PutDatabaseRecord` and `ResultSetRecordSet` do not seem to be covered by unit tests. Should there be additional unit test updates to account for those changes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564482474
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
You are right. I did not notice that `ZoneOffset` extends `ZoneId`. Just focused on that `ZoneId` references an offset and the DST info.
As there is no DST in UTC, it is a simple offset so `ZoneOffset.UTC` will be fine. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564073721
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
Can the `ZoneId.of("UTC")` reference can be replaced with `ZoneOffset.UTC`?
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
##########
@@ -26,12 +27,12 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Array;
+import java.sql.Date;
Review comment:
Changing `java.util.Date` to `java.sql.Date` changes the value of `DATE_CLASS_NAME`, does that have any other implications that should be considered?
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
The method naming and implementation are a little confusing on initial read. The `atZone()` method returns a `ZonedDateTime` and `toLocalDate()` return the `java.time.LocalDate` portion, but does that actually convert the date to the system default time zone? It may be helpful to break this into multiple lines as it is a bit hard to follow.
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
Instead of performing multiple conversions for the test, could this comparison be simplified by comparing `daysSinceEpoch` to the actual number of expected days? That would make the assert much clearer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564218876
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
The code snippet you suggested looks much clearer and avoids the conversion, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564220393
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
+ }
+
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in UTC
+ * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day).
+ *
+ * @param date java.sql.Date with UTC 00:00:00
+ * @return java.sql.Date with local time zone 00:00:00
+ */
+ public static Date convertDateToLocalTZ(Date date) {
+ return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate());
Review comment:
Thanks for the updated suggestion, breaking things out read much more easily, and the `withZoneSameLocal()` method makes more sense now. Based on that snippet, I would suggest just returning the final `Date` object instead of declaring and returning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564220978
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
##########
@@ -26,12 +27,12 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Array;
+import java.sql.Date;
Review comment:
Thanks for responding regarding the column class name handling. I will take another look once you push the next set of updates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564092771
##########
File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
##########
@@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa
throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName);
}
+ /**
+ * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet)
+ * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day).
+ *
+ * @param date java.sql.Date with local time zone 00:00:00
+ * @return java.sql.Date with UTC 00:00:00
+ */
+ public static Date convertDateToUTC(Date date) {
+ return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli());
Review comment:
`atStartOfDay()` has `ZoneId` parameter only.
In my understanding `ZoneOffset` is DST-agnostic, so the `ZoneId` is the right choice in any case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#discussion_r564103224
##########
File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java
##########
@@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio
testConvertToAvroStreamForDateTime(options,
(record, date) -> {
+ java.sql.Date expected = DataTypeUtils.convertDateToUTC(date);
final int daysSinceEpoch = (int) record.get("date");
final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS);
- java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate());
- LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual);
- assertEquals(date, actual);
+ java.sql.Date actual = new java.sql.Date(millisSinceEpoch);
+ LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual);
+ assertEquals(expected, actual);
Review comment:
These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the input of the test case).
I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method.
As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think?
I mean
```
int expectedDaysSinceEpoch = 17444
assertDate.accept(record, expectedDaysSinceEpoch);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on pull request #4781: NIFI-8023: Convert java.sql.Date between UTC/local time zone normaliz…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on pull request #4781:
URL: https://github.com/apache/nifi/pull/4781#issuecomment-767588990
@exceptionfactory Thanks for the review, implemented your suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org