You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "liaotian1005 (via GitHub)" <gi...@apache.org> on 2023/02/09 05:54:11 UTC

[GitHub] [hudi] liaotian1005 opened a new pull request, #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem

liaotian1005 opened a new pull request, #7904:
URL: https://github.com/apache/hudi/pull/7904

   link-hudi write data type of timestamp  format UTC time zone problem.
   
   1. The time zone written by flink is local, but the time zone read is not local
   
   2.flink writes timestamp data, but spark reads different timestamp
   
   Repair:
   
   1. When the cow table and mor table of flink are written, timestamp data is written to the utc time zone by default, which is consistent with utc to rectify the spark reading problem
   
   2. In the cow table, timestamp data is read according to the configuration "read.utc-timezone"
   
   3. In the mor table, the default timestamp read time zone is local.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-5735] Flink-hudi write time format data UTC time zone problem [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-6700106255

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-1423693027

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ac26a880833f1f19aea723f17a13c6efbd86f5ca UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-5735] Flink-hudi write time format data UTC time zone problem [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem
URL: https://github.com/apache/hudi/pull/7904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on pull request #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-1423917806

   There is a similar fix: https://github.com/apache/hudi/pull/7886, there is no need to modify the `TimestampData` because it is timezone aware.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-1424093339

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15055",
       "triggerID" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ac26a880833f1f19aea723f17a13c6efbd86f5ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15055) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-5735] Flink-hudi write time format data UTC time zone problem [hudi]

Posted by "hehuiyuan (via GitHub)" <gi...@apache.org>.
hehuiyuan commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-5423089806

   @danny0405  Teacher Danny, what is the current progress on this PR? We also encountered the same time zone issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-5735] Flink-hudi write time format data UTC time zone problem [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-5423825696

   > @danny0405 Teacher Danny, what is the current progress on this PR? We also encountered the same time zone issue
   
   Flink support the option `write.utc-timezone`, what release did you use?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7904: [HUDI-5735] Flink-hudi write time format data UTC time zone problem

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7904:
URL: https://github.com/apache/hudi/pull/7904#issuecomment-1423697590

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15055",
       "triggerID" : "ac26a880833f1f19aea723f17a13c6efbd86f5ca",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ac26a880833f1f19aea723f17a13c6efbd86f5ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15055) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-5735] Flink-hudi write time format data UTC time zone problem [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #7904:
URL: https://github.com/apache/hudi/pull/7904#discussion_r3861763104


##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetRowDataWriter.java:
##########
@@ -304,6 +304,7 @@ public void write(ArrayData array, int ordinal) {
   }
 
   private long timestampToInt64(TimestampData timestampData) {
+    timestampData = DateUtils.normalLocalDateToUTC(timestampData);
     return utcTimestamp ? timestampData.getMillisecond() : timestampData.toTimestamp().getTime();

Review Comment:
   πŸ€– `normalLocalDateToUTC` is applied unconditionally, but the `!utcTimestamp` branch then calls `timestampData.toTimestamp().getTime()`, which itself applies the localβ†’UTC offset again β€” so when `write.utc-timezone=false` the offset is subtracted twice (same issue in `timestampToInt96` at line 346). Was the normalization meant to run only in the `utcTimestamp` branch?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java:
##########
@@ -43,6 +43,8 @@
 import java.util.List;
 import java.util.Map;
 
+import static org.apache.hudi.util.DateUtils.normalLocalDateToUTC;

Review Comment:
   πŸ€– The new `DateUtils` class appears to be committed without a `.java` extension (`git ls-files` shows `hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/DateUtils`, no `.java`). `javac` only compiles `*.java` files, so this class won't be compiled and every reference to it here and in `ParquetRowDataWriter` would fail to compile. Could you rename the file to `DateUtils.java`?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/AvroToRowDataConverters.java:
##########
@@ -206,16 +206,20 @@ private static AvroToRowDataConverter createMapConverter(LogicalType type) {
   }
 
   private static AvroToRowDataConverter createTimestampConverter(int precision) {
+    return createTimestampConverter(precision,false);

Review Comment:
   πŸ€– The no-arg overload hardcodes `utcTimestamp=false`, and nothing wires the `read.utc-timezone` config into `createTimestampConverter`. So this unconditionally switches the read conversion from `TimestampData.fromInstant(instant)` to `fromTimestamp(Timestamp.from(instant))` for every existing Flink reader β€” a silent local-offset shift on read-back regardless of config. Is the plan to plumb the config through here in a follow-up?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java:
##########
@@ -166,7 +168,8 @@ public Object convert(Schema schema, Object object) {
 
                 @Override
                 public Object convert(Schema schema, Object object) {
-                  return ((TimestampData) object).toInstant().toEpochMilli();
+                  // write timestamp utc timezone 
+                  return normalLocalDateToUTC((TimestampData) object).toInstant().toEpochMilli();

Review Comment:
   πŸ€– nit: the comment `// write timestamp utc timezone ` just restates what `normalLocalDateToUTC` already conveys β€” have you considered removing it (or replacing it with a note explaining *why* this conversion is needed at this specific call site)?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java:
##########


Review Comment:
   πŸ€– **Line 182:** Only the `precision <= 3` branch got the `normalLocalDateToUTC` normalization; this `precision <= 6` (micros) branch still writes without it. Since Flink's default TIMESTAMP precision is 6, the common case is left unconverted, so micros-precision tables won't get the UTC fix and will be inconsistent with the precision-3 path. Should the micros branch be normalized too?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1218,7 +1218,7 @@ protected void setWriteSchemaForDeletes(HoodieTableMetaClient metaClient) {
           throw new HoodieIOException("Latest commit does not have any schema in commit metadata");
         }
       } else {
-        throw new HoodieIOException("Deletes issued without any prior commits");
+        LOG.warn("None rows are deleted because the table is empty");

Review Comment:
   πŸ€– This changes delete-on-empty-table from throwing `HoodieIOException` to just logging a warning, and it lives in engine-agnostic `hudi-client-common`, so it affects Spark and Java clients too, not only Flink. Is that intended scope for a timezone PR? After the warn, `config.setSchema(...)` is skipped β€” could you confirm the downstream delete/commit path handles an unset write schema safely for all engines?
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org