You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2023/06/13 19:55:00 UTC

[GitHub] [flink] snuyanzin opened a new pull request, #22775: [FLINK-17224][table] Support precision for TIME

snuyanzin opened a new pull request, #22775:
URL: https://github.com/apache/flink/pull/22775

   ## What is the purpose of the change
   
   The PR makes support of `TIME` precision (0-3 - currently only this range, I'm going to submit support for micros and nanos in the next PR) including casts, time related functions (`CEIL`, `FLOOR`, `OVERLAPS`, agg `MAX`,  `MIN`,  `LEAD`, `LAG`, singlevalue )
   Also added support of `TIME` precision in csv, json, avro, some tests in Kafka (they use agg `MAX` e.g.)
   
   ## Brief change log
   
   *(for example:)*
     - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
     - *Deployments RPC transmits only the blob storage reference*
     - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   A number of tests in table-planner, avro, csv, json, kafka dealing with `TIME`
   
   especially `flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java` and `flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java`
   checking different variations of casts to and from `TIME`
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: ( no)
     - The runtime per-record code paths (performance sensitive): ( no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
     - The S3 file system connector: ( no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? ( no)
     - If yes, how is the feature documented? (not applicable )
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-2172858716

   Thanks for your feedback
   >Could we change sth so that we don't lose data when a precision >3 is used? Could we throw an exception somewhere explicitly?
   
   This seems a questionable thing how Flink should behave.
   
   I checked it for Postgres where for `TIME` the precision `>6` is not supported and Postgres continues working with loosing data   (it puts warning in it's output log about that)
   
   On the other side MySQL throws an exception.
   
   I checked Flink SQL for
   ```sql
   SELECT CAST('1111-11-11 11:11:11.123456789123' AS timestamp(12))
   ```
   
   and it fails as 
   ```
   Caused by: java.time.format.DateTimeParseException: Text '1111-11-11 11:11:11.123456789123' could not be parsed, unparsed text found at index 29
   	at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
   	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
   	at org.apache.flink.table.utils.DateTimeUtils.parseTimestampData(DateTimeUtils.java:413)
   	at org.apache.flink.table.data.binary.BinaryStringDataUtil.toTimestamp(BinaryStringDataUtil.java:612)
   ```
   
   I think we could add similar behavior for `TIME`	


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-2172834841

   Rebased to solve the conflicts 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22775: [FLINK-17224][table] Support precision for TIME
URL: https://github.com/apache/flink/pull/22775


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #22775: [FLINK-17224][table] Support precision for TIME

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-1589940884

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c8b5a3ac0ad29fd4173fc7edd68e27762a59d6a4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c8b5a3ac0ad29fd4173fc7edd68e27762a59d6a4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c8b5a3ac0ad29fd4173fc7edd68e27762a59d6a4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #22775:
URL: https://github.com/apache/flink/pull/22775#discussion_r1627434353


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -763,6 +763,68 @@ private static List<TestSetSpec> allTypesBasic() {
                         // ROW
                         // RAW
                         .build(),
+                CastTestSpecBuilder.testCastTo(TIME(2))
+                        .fromCase(TIME(), null, null)
+                        .failRuntime(CHAR(3), "foo", DateTimeException.class)
+                        .failRuntime(VARCHAR(5), "Flink", DateTimeException.class)
+                        .failRuntime(STRING(), "Flink", DateTimeException.class)
+                        .fromCase(STRING(), "123", LocalTime.of(23, 0, 0))
+                        .fromCase(STRING(), "123:45", LocalTime.of(23, 45, 0))
+                        .failRuntime(STRING(), "2021-09-27", DateTimeException.class)
+                        .failRuntime(STRING(), "2021-09-27 12:34:56", DateTimeException.class)

Review Comment:
   I am wondering if we should duplicate those. Internally, there is no difference if we cast those cases to `TIME(2)` or `TIME(0)`.
   
   Personally, I'd be leaning to leave only the cases where we truncate the precision.



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -763,6 +763,68 @@ private static List<TestSetSpec> allTypesBasic() {
                         // ROW
                         // RAW
                         .build(),
+                CastTestSpecBuilder.testCastTo(TIME(2))
+                        .fromCase(TIME(), null, null)
+                        .failRuntime(CHAR(3), "foo", DateTimeException.class)
+                        .failRuntime(VARCHAR(5), "Flink", DateTimeException.class)
+                        .failRuntime(STRING(), "Flink", DateTimeException.class)
+                        .fromCase(STRING(), "123", LocalTime.of(23, 0, 0))
+                        .fromCase(STRING(), "123:45", LocalTime.of(23, 45, 0))
+                        .failRuntime(STRING(), "2021-09-27", DateTimeException.class)
+                        .failRuntime(STRING(), "2021-09-27 12:34:56", DateTimeException.class)
+                        .fromCase(
+                                STRING(),
+                                "12:34:56.123456789",
+                                LocalTime.of(12, 34, 56, 120_000_000))
+                        .failRuntime(
+                                STRING(), "2021-09-27 12:34:56.123456789", DateTimeException.class)
+                        // Not supported - no fix
+                        .failValidation(BOOLEAN(), true)
+                        .failTableApiValidation(BINARY(2), DEFAULT_BINARY)
+                        .failTableApiValidation(VARBINARY(5), DEFAULT_VARBINARY)
+                        .failTableApiValidation(BYTES(), DEFAULT_BYTES)
+                        .failValidation(DECIMAL(5, 3), 12.345)
+                        .failValidation(TINYINT(), DEFAULT_POSITIVE_TINY_INT)
+                        .failValidation(SMALLINT(), DEFAULT_POSITIVE_SMALL_INT)
+                        .failValidation(INT(), DEFAULT_POSITIVE_INT)
+                        .failValidation(BIGINT(), DEFAULT_POSITIVE_BIGINT)
+                        .failValidation(FLOAT(), DEFAULT_POSITIVE_FLOAT)
+                        .failValidation(DOUBLE(), DEFAULT_POSITIVE_DOUBLE)
+                        .failValidation(DATE(), DEFAULT_DATE)
+                        //
+                        .fromCase(TIME(5), DEFAULT_TIME, LocalTime.of(12, 34, 56, 120_000_000))

Review Comment:
   Shouldn't this fail? Can we handle `TIME` with precision of `5`?



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -1038,9 +1112,9 @@ private static List<TestSetSpec> toStringCasts() {
                                 DEFAULT_NEGATIVE_DOUBLE,
                                 String.valueOf(DEFAULT_NEGATIVE_DOUBLE))
                         .fromCase(DATE(), DEFAULT_DATE, "2021-09-24")
-                        // https://issues.apache.org/jira/browse/FLINK-17224 Currently, fractional
-                        // seconds are lost
-                        .fromCase(TIME(5), DEFAULT_TIME, "12:34:56")
+                        .fromCase(TIME(5), DEFAULT_TIME, "12:34:56.123")

Review Comment:
   This shows, sth is wrong with `TIME(5)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] snuyanzin commented on pull request #22775: [FLINK-17224][table] Support precision for TIME

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-1590481608

   @dawidwys since you are the reporter of the issue could you please have a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-2774618880

   This PR has been closed since it has not had any activity in 120 days. 
   If you feel like this was a mistake, or you would like to continue working on it, 
   please feel free to re-open the PR and ask for a review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22775:
URL: https://github.com/apache/flink/pull/22775#discussion_r1642482796


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -1038,9 +1112,9 @@ private static List<TestSetSpec> toStringCasts() {
                                 DEFAULT_NEGATIVE_DOUBLE,
                                 String.valueOf(DEFAULT_NEGATIVE_DOUBLE))
                         .fromCase(DATE(), DEFAULT_DATE, "2021-09-24")
-                        // https://issues.apache.org/jira/browse/FLINK-17224 Currently, fractional
-                        // seconds are lost
-                        .fromCase(TIME(5), DEFAULT_TIME, "12:34:56")
+                        .fromCase(TIME(5), DEFAULT_TIME, "12:34:56.123")

Review Comment:
   May be we still should put comment like micro and nano are not supported for `TIME` yet?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-17224][table] Support precision for TIME [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22775:
URL: https://github.com/apache/flink/pull/22775#issuecomment-2595359595

   This PR is being marked as stale since it has not had any activity in the last 180 days. 
   If you would like to keep this PR alive, please leave a comment asking for a review. 
   If the PR has merge conflicts, update it with the latest from the base branch.
   
   If you are having difficulty finding a reviewer, please reach out to the 
   community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
   
   If this PR is no longer valid or desired, please feel free to close it. 
   If no activity occurs in the next 90 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org