You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/31 12:21:17 UTC
[GitHub] [flink] alpreu opened a new pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
alpreu opened a new pull request #19302:
URL: https://github.com/apache/flink/pull/19302
## What is the purpose of the change
Dynamic indexes generated by the `IndexGeneratorFactory` ignore the local timezone and emit UTC timezone information instead. This PR fixes the issue by using the local timezone instead.
This PR will also need to be backported to 1.15
## Brief change log
- Changed behavior for Sink-based and SinkFunction-based connector
- Added/Adjusted tests
## Verifying this change
This change is already covered by existing tests, such as `testDynamicIndexDefaultFormatTimestampWithLocalTimeZone` and the newly introduced test
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hackergin commented on a change in pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
hackergin commented on a change in pull request #19302:
URL: https://github.com/apache/flink/pull/19302#discussion_r840165388
##########
File path: flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##########
@@ -186,7 +185,10 @@ private static DynamicFormatter createFormatFunction(
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
return (value, dateTimeFormatter) -> {
TimestampData indexField = (TimestampData) value;
- return indexField.toInstant().atZone(ZoneOffset.UTC).format(dateTimeFormatter);
+ return indexField
+ .toInstant()
Review comment:
When `createIndexGenerator` there is already a localTimeZoneId , So we don't need to get is from `ZoneId.systemDefault()` ?
##########
File path: flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/table/IndexGeneratorFactory.java
##########
@@ -183,7 +182,10 @@ private static DynamicFormatter createFormatFunction(
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
return (value, dateTimeFormatter) -> {
TimestampData indexField = (TimestampData) value;
- return indexField.toInstant().atZone(ZoneOffset.UTC).format(dateTimeFormatter);
+ return indexField
+ .toInstant()
Review comment:
the same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b0b0152e957da57276f6e79f0fa7f50fcf8794e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
}, {
"hash" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b0b0152e957da57276f6e79f0fa7f50fcf8794e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041)
* d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
}, {
"hash" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34090",
"triggerID" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34090)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b0b0152e957da57276f6e79f0fa7f50fcf8794e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b0b0152e957da57276f6e79f0fa7f50fcf8794e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpreu commented on a change in pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
alpreu commented on a change in pull request #19302:
URL: https://github.com/apache/flink/pull/19302#discussion_r840342878
##########
File path: flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/table/IndexGeneratorFactory.java
##########
@@ -186,7 +185,10 @@ private static DynamicFormatter createFormatFunction(
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
return (value, dateTimeFormatter) -> {
TimestampData indexField = (TimestampData) value;
- return indexField.toInstant().atZone(ZoneOffset.UTC).format(dateTimeFormatter);
+ return indexField
+ .toInstant()
Review comment:
You're right, I now changed it to reuse the localTimeZoneId we pass in there 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19302: [FLINK-26810][connectors/elasticsearch] Use local timezone for TIMESTAMP_WITH_LOCAL_TIMEZONE fields in dynamic index
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19302:
URL: https://github.com/apache/flink/pull/19302#issuecomment-1084520142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041",
"triggerID" : "8b0b0152e957da57276f6e79f0fa7f50fcf8794e",
"triggerType" : "PUSH"
}, {
"hash" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34090",
"triggerID" : "d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b0b0152e957da57276f6e79f0fa7f50fcf8794e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34041)
* d69dc5ebcf0a5b4dc02532d697f0e0aea65e9b77 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34090)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org