You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/18 02:18:23 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
nsivabalan opened a new pull request #4024:
URL: https://github.com/apache/hudi/pull/4024
## What is the purpose of the pull request
- Redo of https://github.com/apache/hudi/pull/3824.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975572571
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975569869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975703379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
}, {
"hash" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595",
"triggerID" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#discussion_r752823452
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -33,14 +35,27 @@
*/
public class HoodieInstantTimeGenerator {
// Format of the timestamp used for an Instant
- private static final String INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
- private static final int INSTANT_TIMESTAMP_FORMAT_LENGTH = INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String SECS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
+ public static final int SECS_INSTANT_ID_LENGTH = SECS_INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmssSSS";
+ public static final int MILLIS_INSTANT_ID_LENGTH = MILLIS_INSTANT_TIMESTAMP_FORMAT.length();
+ public static final int MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH = MILLIS_INSTANT_TIMESTAMP_FORMAT.length();
// Formatter to generate Instant timestamps
- private static DateTimeFormatter INSTANT_TIME_FORMATTER = DateTimeFormatter.ofPattern(INSTANT_TIMESTAMP_FORMAT);
+ // Unfortunately millisecond format is not parsable as is https://bugs.openjdk.java.net/browse/JDK-8031085. hence have to do appendValue()
+ private static DateTimeFormatter MILLIS_INSTANT_TIME_FORMATTER = new DateTimeFormatterBuilder().appendPattern(SECS_INSTANT_TIMESTAMP_FORMAT)
+ .appendValue(ChronoField.MILLI_OF_SECOND, 3).toFormatter();
+ private static final String MILLIS_GRANULARITY_DATE_FORMAT = "yyyy-MM-dd HH:mm:ss:SSS";
Review comment:
usually sec and millis were separated by `.` instead of `:`
`yyyy-MM-dd HH:mm:ss.SSS` is a more common format
some examples in https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975660279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
}, {
"hash" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595",
"triggerID" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
* 30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975475103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972451341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973233000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973235040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972478317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975475103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975654133
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972443067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975715501
@xushiyan : I am going to land this now. But, can you update your test results here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972478317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975467898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
* 61eb78b61845964a94cd0304aeeb8be954323f0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4024:
URL: https://github.com/apache/hudi/pull/4024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#discussion_r752008715
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -33,14 +35,27 @@
*/
public class HoodieInstantTimeGenerator {
// Format of the timestamp used for an Instant
- private static final String INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
- private static final int INSTANT_TIMESTAMP_FORMAT_LENGTH = INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String SECS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
+ public static final int SECS_INSTANT_ID_LENGTH = SECS_INSTANT_TIMESTAMP_FORMAT.length();
+ private static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmssSSS";
Review comment:
Can we actually introduce `.` in the time format? it improves readability and people can easily distinguish it from the old format. Plus it will bypass the jdk bug with millisecond parsing?
```suggestion
private static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss.SSS";
```
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -72,24 +72,24 @@
protected HoodieTableMetaClient metaClient;
/**
- * Parse the timestamp of an Instant and return a {@code SimpleDateFormat}.
+ * Parse the timestamp of an Instant and return a {@code Date}.
*/
- public static Date parseInstantTime(String timestamp) throws ParseException {
- return HoodieInstantTimeGenerator.parseInstantTime(timestamp);
+ public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
+ return HoodieInstantTimeGenerator.parseDateFromInstantTime(timestamp);
}
/**
* Format the java.time.Instant to a String representing the timestamp of a Hoodie Instant.
*/
- public static String formatInstantTime(Instant timestamp) {
- return HoodieInstantTimeGenerator.formatInstantTime(timestamp);
+ public static String getInstantForDate(Instant timestamp) {
+ return HoodieInstantTimeGenerator.getInstantForDate(timestamp);
Review comment:
`getInstantForDate()` sounds like taking a `Date` and returning a java `Instant`. I think the original name sounds just fine.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -52,36 +67,65 @@ public static String createNewInstantTime(long milliseconds) {
String newCommitTime;
do {
Date d = new Date(System.currentTimeMillis() + milliseconds);
- newCommitTime = INSTANT_TIME_FORMATTER.format(convertDateToTemporalAccessor(d));
+ newCommitTime = MILLIS_INSTANT_TIME_FORMATTER.format(convertDateToTemporalAccessor(d));
} while (HoodieTimeline.compareTimestamps(newCommitTime, HoodieActiveTimeline.LESSER_THAN_OR_EQUALS, oldVal));
return newCommitTime;
});
}
- public static Date parseInstantTime(String timestamp) throws ParseException {
+ public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
try {
- LocalDateTime dt = LocalDateTime.parse(timestamp, INSTANT_TIME_FORMATTER);
+ // Enables backwards compatibility with non-millisecond granularity instants
+ String timestampInMillis = timestamp;
+ if (isSecondGranularity(timestamp)) {
+ // Add milliseconds to the instant in order to parse successfully
+ timestampInMillis = timestamp + DEFAULT_MILLIS_EXT;
+ } else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) {
+ // compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with secs granularity
+ timestampInMillis = timestamp.substring(0, MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH);
+ }
+
+ LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER);
return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant());
} catch (DateTimeParseException e) {
// Special handling for all zero timestamp which is not parsable by DateTimeFormatter
if (timestamp.equals(ALL_ZERO_TIMESTAMP)) {
return new Date(0);
}
- // compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with secs granularity
- if (timestamp.length() > INSTANT_TIMESTAMP_FORMAT_LENGTH) {
- LocalDateTime dt = LocalDateTime.parse(timestamp.substring(0, INSTANT_TIMESTAMP_FORMAT_LENGTH), INSTANT_TIME_FORMATTER);
- return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant());
- }
throw e;
}
}
- public static String formatInstantTime(Instant timestamp) {
- return INSTANT_TIME_FORMATTER.format(timestamp);
+ private static boolean isSecondGranularity(String instant) {
+ return instant.length() == SECS_INSTANT_ID_LENGTH;
+ }
+
+ public static String getInstantForDate(Instant timestamp) {
Review comment:
i just felt `getInstantForXXX()` confuses people as it sounds like returning java Instant. I think `formatInstantTime()`, `formatDate()` suit better, at least people expect it to return String.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -72,24 +72,24 @@
protected HoodieTableMetaClient metaClient;
/**
- * Parse the timestamp of an Instant and return a {@code SimpleDateFormat}.
+ * Parse the timestamp of an Instant and return a {@code Date}.
*/
- public static Date parseInstantTime(String timestamp) throws ParseException {
- return HoodieInstantTimeGenerator.parseInstantTime(timestamp);
+ public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
+ return HoodieInstantTimeGenerator.parseDateFromInstantTime(timestamp);
}
/**
* Format the java.time.Instant to a String representing the timestamp of a Hoodie Instant.
*/
- public static String formatInstantTime(Instant timestamp) {
- return HoodieInstantTimeGenerator.formatInstantTime(timestamp);
+ public static String getInstantForDate(Instant timestamp) {
+ return HoodieInstantTimeGenerator.getInstantForDate(timestamp);
}
/**
* Format the Date to a String representing the timestamp of a Hoodie Instant.
*/
- public static String formatInstantTime(Date timestamp) {
- return HoodieInstantTimeGenerator.formatInstantTime(timestamp);
+ public static String getInstantForDate(Date timestamp) {
+ return HoodieInstantTimeGenerator.getInstantForDate(timestamp);
Review comment:
similarly how about `formatDate()`.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -33,14 +35,27 @@
*/
public class HoodieInstantTimeGenerator {
// Format of the timestamp used for an Instant
- private static final String INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
- private static final int INSTANT_TIMESTAMP_FORMAT_LENGTH = INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String SECS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
+ public static final int SECS_INSTANT_ID_LENGTH = SECS_INSTANT_TIMESTAMP_FORMAT.length();
+ private static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmssSSS";
Review comment:
why not public? also feel the constants below can be all public final. don't see harm exposing them but convenience in testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973004707
Tested the upgrade form 090 to this patch. a failed clustering was able to successfully complete after upgrade. And new commits are in ms format.
![Uploading Screen Shot 2021-11-18 at 11.02.22 AM.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973233000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975467898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
* 61eb78b61845964a94cd0304aeeb8be954323f0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975657617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
}, {
"hash" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
* 30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975470288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972443067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973039009
> I can add a config, but would like to make the default true. with multi-writer, there could be clashes among multiple writers for same commit time. and so this is a much needed fix. If we run into issues, probably we can flip using the config. give me sometime, I need to fix the patch. bcoz, we also need to consider enabling the config and then disabling it. so, have to support both code paths and writers and readers.
Ok if it's about avoiding conflicts, do you need more granularity? all the way to nanoseconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975470288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973284240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975660279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
}, {
"hash" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595",
"triggerID" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
* 30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3595)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975636307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975572571
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559][WIP] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-972451341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#discussion_r752358322
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -33,14 +35,27 @@
*/
public class HoodieInstantTimeGenerator {
// Format of the timestamp used for an Instant
- private static final String INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
- private static final int INSTANT_TIMESTAMP_FORMAT_LENGTH = INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String SECS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
+ public static final int SECS_INSTANT_ID_LENGTH = SECS_INSTANT_TIMESTAMP_FORMAT.length();
+ private static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmssSSS";
Review comment:
@vinothchandar : any thoughts here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973284240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#discussion_r752823452
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -33,14 +35,27 @@
*/
public class HoodieInstantTimeGenerator {
// Format of the timestamp used for an Instant
- private static final String INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
- private static final int INSTANT_TIMESTAMP_FORMAT_LENGTH = INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String SECS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmss";
+ public static final int SECS_INSTANT_ID_LENGTH = SECS_INSTANT_TIMESTAMP_FORMAT.length();
+ public static final String MILLIS_INSTANT_TIMESTAMP_FORMAT = "yyyyMMddHHmmssSSS";
+ public static final int MILLIS_INSTANT_ID_LENGTH = MILLIS_INSTANT_TIMESTAMP_FORMAT.length();
+ public static final int MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH = MILLIS_INSTANT_TIMESTAMP_FORMAT.length();
// Formatter to generate Instant timestamps
- private static DateTimeFormatter INSTANT_TIME_FORMATTER = DateTimeFormatter.ofPattern(INSTANT_TIMESTAMP_FORMAT);
+ // Unfortunately millisecond format is not parsable as is https://bugs.openjdk.java.net/browse/JDK-8031085. hence have to do appendValue()
+ private static DateTimeFormatter MILLIS_INSTANT_TIME_FORMATTER = new DateTimeFormatterBuilder().appendPattern(SECS_INSTANT_TIMESTAMP_FORMAT)
+ .appendValue(ChronoField.MILLI_OF_SECOND, 3).toFormatter();
+ private static final String MILLIS_GRANULARITY_DATE_FORMAT = "yyyy-MM-dd HH:mm:ss:SSS";
Review comment:
usually sec and millis were separated by `.` instead of `:`
`yyyy-MM-dd HH:mm:ss.SSS` is a more common format
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -431,6 +431,12 @@
.sinceVersion("0.10.0")
.withDocumentation("File Id Prefix provider class, that implements `org.apache.hudi.fileid.FileIdPrefixProvider`");
+ public static final ConfigProperty<Boolean> ENABLE_MILLIS_COMMIT_TIMESTAMP_FORMAT = ConfigProperty
+ .key("hoodie.enable.millisecs.commit.timestamp.format")
+ .defaultValue(true)
+ .sinceVersion("0.10.0")
+ .withDocumentation("Enables millisecs granularity for commit times. Falls back to secs level granularity if disabled.");
Review comment:
still need this?
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/table/timeline/TestHoodieActiveTimeline.java
##########
@@ -476,16 +476,37 @@ public void testCreateNewInstantTime() throws Exception {
@Test
public void testMetadataCompactionInstantDateParsing() throws ParseException {
// default second granularity instant ID
- String secondGranularityInstant = "20210101120101";
- Date defaultSecsGranularityDate = HoodieActiveTimeline.parseInstantTime(secondGranularityInstant);
+ String secondGranularityInstant = "20210101120101123";
+ Date defaultSecsGranularityDate = HoodieActiveTimeline.parseDateFromInstantTime(secondGranularityInstant);
Review comment:
```suggestion
String msGranularityInstant = "20210101120101123";
Date defaultMsGranularityDate = HoodieActiveTimeline.parseDateFromInstantTime(msGranularityInstant);
```
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
##########
@@ -52,36 +67,65 @@ public static String createNewInstantTime(long milliseconds) {
String newCommitTime;
do {
Date d = new Date(System.currentTimeMillis() + milliseconds);
- newCommitTime = INSTANT_TIME_FORMATTER.format(convertDateToTemporalAccessor(d));
+ newCommitTime = MILLIS_INSTANT_TIME_FORMATTER.format(convertDateToTemporalAccessor(d));
} while (HoodieTimeline.compareTimestamps(newCommitTime, HoodieActiveTimeline.LESSER_THAN_OR_EQUALS, oldVal));
return newCommitTime;
});
}
- public static Date parseInstantTime(String timestamp) throws ParseException {
+ public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
try {
- LocalDateTime dt = LocalDateTime.parse(timestamp, INSTANT_TIME_FORMATTER);
+ // Enables backwards compatibility with non-millisecond granularity instants
+ String timestampInMillis = timestamp;
+ if (isSecondGranularity(timestamp)) {
+ // Add milliseconds to the instant in order to parse successfully
+ timestampInMillis = timestamp + DEFAULT_MILLIS_EXT;
+ } else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) {
+ // compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with secs granularity
Review comment:
```suggestion
// compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with millis granularity
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973226363
@xushiyan : I started working on adding in the config. Looks like it might be involved one. bcoz, as of now, creating new instant time is static. but now, if we introduce a config, we have to pass in the writeConfig/TableConfig everywhere. some code paths may not even access to both or either of them. Wondering if we really need to.
I am ok spending time, but looking to get your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973009464
I can add a config, but would like to make the default true. with multi-writer, there could be clashes among multiple writers for same commit time. and so this is a much needed fix. If we run into issues, probably we can flip using the config.
give me sometime, I need to fix the patch. bcoz, we also need to consider enabling the config and then disabling it. so, have to support both code paths and writers and readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-973235040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 497fd871304a7696df82e2e89d79c49b92308a18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450)
* 3f4f4d147d0bafd5305b1e6dd141116e2790591b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975657617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
}, {
"hash" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
* 30aa21542b45a77c9a678b8e0d2f5a0d1c1f4646 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975636307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4024: [HUDI-2559] Converting commit timestamp format to millisecs
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4024:
URL: https://github.com/apache/hudi/pull/4024#issuecomment-975569869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "497fd871304a7696df82e2e89d79c49b92308a18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3450",
"triggerID" : "497fd871304a7696df82e2e89d79c49b92308a18",
"triggerType" : "PUSH"
}, {
"hash" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3477",
"triggerID" : "3f4f4d147d0bafd5305b1e6dd141116e2790591b",
"triggerType" : "PUSH"
}, {
"hash" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584",
"triggerID" : "61eb78b61845964a94cd0304aeeb8be954323f0b",
"triggerType" : "PUSH"
}, {
"hash" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "acb91ee591874b835f7e7d258e7be22ed4ee9cc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61eb78b61845964a94cd0304aeeb8be954323f0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3584)
* acb91ee591874b835f7e7d258e7be22ed4ee9cc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org