You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/13 06:35:44 UTC
[GitHub] [hudi] scxwhite opened a new pull request, #7664: [HUDI-5551] support seconds unit on event_time
scxwhite opened a new pull request, #7664:
URL: https://github.com/apache/hudi/pull/7664
### Change Logs
<img width="1039" alt="image" src="https://user-images.githubusercontent.com/23207189/212252797-fcca4d17-d49b-48c0-970c-a576eaeed8ea.png">
When I used grafana to draw the metrics of commitFreshnessInMs, I found that the value of some tables was particularly large, and the tracking code found that it was the event_time unit of time is seconds, so submitting this pr supports the event_time in seconds
### Impact
event_time metrics
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1385019069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
}, {
"hash" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354",
"triggerID" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"triggerType" : "PUSH"
}, {
"hash" : "674eef810f4f188aaf0f505189674e454186e208",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "674eef810f4f188aaf0f505189674e454186e208",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302)
* 7fa0b38ff13bce16a12b35a9f009b414854c9fe6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354)
* 674eef810f4f188aaf0f505189674e454186e208 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] scxwhite commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by "scxwhite (via GitHub)" <gi...@apache.org>.
scxwhite commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1411553653
@danny0405 @yihua The test has been added and completed, please help to review it, thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7664:
URL: https://github.com/apache/hudi/pull/7664#discussion_r1069132028
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -101,7 +101,18 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
String eventTimeVal = optionalRecordMetadata.get().getOrDefault(METADATA_EVENT_TIME_KEY, null);
try {
if (!StringUtils.isNullOrEmpty(eventTimeVal)) {
- long eventTime = DateTimeUtils.parseDateTime(eventTimeVal).toEpochMilli();
+ int length = eventTimeVal.length();
+ long millisEventTime;
+ // eventTimeVal in seconds unit
+ if (length == 10) {
+ millisEventTime = Long.parseLong(eventTimeVal) * 1000;
+ } else if (length == 13) {
+ // eventTimeVal in millis unit
+ millisEventTime = Long.parseLong(eventTimeVal);
+ } else {
+ throw new IllegalArgumentException("not support event_time format:" + eventTimeVal);
Review Comment:
Is it reasonable to throw directly here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1381433123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7664:
URL: https://github.com/apache/hudi/pull/7664#discussion_r1069132028
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -101,7 +101,18 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
String eventTimeVal = optionalRecordMetadata.get().getOrDefault(METADATA_EVENT_TIME_KEY, null);
try {
if (!StringUtils.isNullOrEmpty(eventTimeVal)) {
- long eventTime = DateTimeUtils.parseDateTime(eventTimeVal).toEpochMilli();
+ int length = eventTimeVal.length();
+ long millisEventTime;
+ // eventTimeVal in seconds unit
+ if (length == 10) {
+ millisEventTime = Long.parseLong(eventTimeVal) * 1000;
+ } else if (length == 13) {
+ // eventTimeVal in millis unit
+ millisEventTime = Long.parseLong(eventTimeVal);
+ } else {
+ throw new IllegalArgumentException("not support event_time format:" + eventTimeVal);
Review Comment:
Is it reasonable to throw directly here ? And if possible, can we add some tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] scxwhite commented on a diff in pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
scxwhite commented on code in PR #7664:
URL: https://github.com/apache/hudi/pull/7664#discussion_r1069151140
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -101,7 +101,18 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
String eventTimeVal = optionalRecordMetadata.get().getOrDefault(METADATA_EVENT_TIME_KEY, null);
try {
if (!StringUtils.isNullOrEmpty(eventTimeVal)) {
- long eventTime = DateTimeUtils.parseDateTime(eventTimeVal).toEpochMilli();
+ int length = eventTimeVal.length();
+ long millisEventTime;
+ // eventTimeVal in seconds unit
+ if (length == 10) {
+ millisEventTime = Long.parseLong(eventTimeVal) * 1000;
+ } else if (length == 13) {
+ // eventTimeVal in millis unit
+ millisEventTime = Long.parseLong(eventTimeVal);
+ } else {
+ throw new IllegalArgumentException("not support event_time format:" + eventTimeVal);
Review Comment:
> Is it reasonable to throw directly here ? And if possible, can we add some tests.
Thanks for your review.
This code is in try catch block. When the user's log level is set to debug, you can see these exception logs.
I will try to add some tests next week.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -101,7 +101,18 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
String eventTimeVal = optionalRecordMetadata.get().getOrDefault(METADATA_EVENT_TIME_KEY, null);
try {
if (!StringUtils.isNullOrEmpty(eventTimeVal)) {
- long eventTime = DateTimeUtils.parseDateTime(eventTimeVal).toEpochMilli();
+ int length = eventTimeVal.length();
+ long millisEventTime;
+ // eventTimeVal in seconds unit
+ if (length == 10) {
+ millisEventTime = Long.parseLong(eventTimeVal) * 1000;
+ } else if (length == 13) {
+ // eventTimeVal in millis unit
+ millisEventTime = Long.parseLong(eventTimeVal);
+ } else {
+ throw new IllegalArgumentException("not support event_time format:" + eventTimeVal);
Review Comment:
> Is it reasonable to throw directly here ? And if possible, can we add some tests.
Thanks for your review.
This code is in try catch block. When the user's log level is set to debug, you can see these exception logs.
I will try to add some tests next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #7664:
URL: https://github.com/apache/hudi/pull/7664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1385494598
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
}, {
"hash" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354",
"triggerID" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"triggerType" : "PUSH"
}, {
"hash" : "674eef810f4f188aaf0f505189674e454186e208",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14358",
"triggerID" : "674eef810f4f188aaf0f505189674e454186e208",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 674eef810f4f188aaf0f505189674e454186e208 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14358)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1381843211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1384916684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
}, {
"hash" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302)
* 7fa0b38ff13bce16a12b35a9f009b414854c9fe6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1381427203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1385026602
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
}, {
"hash" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354",
"triggerID" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"triggerType" : "PUSH"
}, {
"hash" : "674eef810f4f188aaf0f505189674e454186e208",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14358",
"triggerID" : "674eef810f4f188aaf0f505189674e454186e208",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fa0b38ff13bce16a12b35a9f009b414854c9fe6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354)
* 674eef810f4f188aaf0f505189674e454186e208 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14358)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7664:
URL: https://github.com/apache/hudi/pull/7664#issuecomment-1384922271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302",
"triggerID" : "2f4ee14477c6868151f3d14eb1f3535d3eafb11d",
"triggerType" : "PUSH"
}, {
"hash" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354",
"triggerID" : "7fa0b38ff13bce16a12b35a9f009b414854c9fe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14302)
* 7fa0b38ff13bce16a12b35a9f009b414854c9fe6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org