You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/04 09:57:43 UTC

[GitHub] [hudi] YannByron opened a new pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

YannByron opened a new pull request #4749:
URL: https://github.com/apache/hudi/pull/4749


   
   Hoodie converts `Timestamp` to TIMESTAMP_MICROS format when upsert and other operations, except `bulk_insert`.
   
   And `bulk_insert` enables `hoodie.datasource.write.row.writer.enable`, and use `HoodieRowParquetWriteSupport` to write datas. 
   
   For the issue #4552 , that will cause problems by default. So i suggest to modify the `hoodie.parquet.outputtimestamptype` default value to TIMESTAMP_MICROS so that it will be convenience to users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YannByron edited a comment on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
YannByron edited a comment on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1031018797


   @nsivabalan
   For now this config `hoodie.parquet.outputtimestamptype` is only used in 'bulk_insert' mode, not `upsert`. and it won't affect timestamp based partition path.
   This config will decide that Long values representing the timestamp type will be persisted in parquet file using the format of `TIMESTAMP_MILLIS ` or `TIMESTAMP_MICROS` when `bulk_insert`. Like `2021-10-26 03:44:03`, converting it to Long type is `1635234243` in `TIMESTAMP_MILLIS ` format, but `1635234243000` in `TIMESTAMP_MICROS` format.
   BTW, `upsert` uses the `TIMESTAMP_MICROS` format. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YannByron commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
YannByron commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1031286560


   @codope 
   sorry for the confused description. I edited again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1031319868


   > sorry for the confused description. I edited again.
   
   Thanks for the clarification. 
   I still have this question. Suppose user did bulk insert in a table created with version 0.10.0 and now upgrades and runs another bulk insert with 0.11.0 where default is now TIMESTAMP_MICROS. Is the type change from TIMESTAMP_MILLIS to TIMESTAMP_MICROS backward compatible? For your example, in the newer version, `1635234243` would be some date in the year 1970 while `1635234243000` would be much more recent. Wouldn't this affect the queries? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1033159091


   thanks Sagar for bringing up a good point. Just playing devil's advocate. Spark in general has much incompatability issues even after being very mature. So, for the benefit we get (same behavior across bulk insert and other operations), I am thinking if we can go ahead with this change. Again, my assumption is that w/ non row writer operations, we are already honoring micro secs with timestamp type. 
   
   Even today, users have some inconsistencies here which they are living with. 
   especially given this is not affecting partition path flow. just some random timestamp type fields. @YannByron : correct me if I am mistaken in any of the above statements. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1032708222


   can we file a jira please as we try to get consensus


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1029824956


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1495c27218d6f36020416563786f37c29f7b3421",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731",
       "triggerID" : "1495c27218d6f36020416563786f37c29f7b3421",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1495c27218d6f36020416563786f37c29f7b3421 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1029824956


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1495c27218d6f36020416563786f37c29f7b3421",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731",
       "triggerID" : "1495c27218d6f36020416563786f37c29f7b3421",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1495c27218d6f36020416563786f37c29f7b3421 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1029886905


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1495c27218d6f36020416563786f37c29f7b3421",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731",
       "triggerID" : "1495c27218d6f36020416563786f37c29f7b3421",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1495c27218d6f36020416563786f37c29f7b3421 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5731) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YannByron commented on pull request #4749: [HUDI-3402] Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
YannByron commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1033568428


   > can we file a jira please as we try to get consensus
   
   https://issues.apache.org/jira/browse/HUDI-3402


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1029822444


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1495c27218d6f36020416563786f37c29f7b3421",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1495c27218d6f36020416563786f37c29f7b3421",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1495c27218d6f36020416563786f37c29f7b3421 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YannByron commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
YannByron commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1031018797


   @nsivabalan
   For now this config `hoodie.parquet.outputtimestamptype` is only used in 'bulk_insert' mode, not `upsert`. and it won't affect timestamp based partition path.
   If set false, Long values representing the timestamp type will be persisted in parquet file using the format of `TIMESTAMP_MILLIS ` when `bulk_insert`. And set true, using the format of `TIMESTAMP_MICROS`.
   BTW, `upsert` uses the `TIMESTAMP_MICROS` format. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #4749: [HUDI-3402] Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4749:
URL: https://github.com/apache/hudi/pull/4749


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1031272111


   > If set false, Long values representing the timestamp type will be persisted in parquet file using the format of `TIMESTAMP_MILLIS ` when `bulk_insert`. And set true, using the format of `TIMESTAMP_MICROS`. BTW, `upsert` uses the `TIMESTAMP_MICROS` format.
   
   Which config is being set to true/false in the above statement?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #4749: [HUDI-3402] Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1036439616


   cool, thanks a lot man! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4749: Set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS by default

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4749:
URL: https://github.com/apache/hudi/pull/4749#issuecomment-1029822444


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1495c27218d6f36020416563786f37c29f7b3421",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1495c27218d6f36020416563786f37c29f7b3421",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1495c27218d6f36020416563786f37c29f7b3421 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org