You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bingquanzhao (via GitHub)" <gi...@apache.org> on 2024/07/10 02:39:42 UTC
[PR] [Fix](source) Fixed incorrect Map value reading when key/value is of DATE or DATETIME type [doris-flink-connector]
bingquanzhao opened a new pull request, #419:
URL: https://github.com/apache/doris-flink-connector/pull/419
Fixed incorrect Map value reading when key/value is of DATE or DATETIME type.
When encountering the Date type, arrow returns a DateDay Integer, and when encountering the datetime type, arrow returns timestamps of varying precision, including seconds, milliseconds, and microseconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](source) Fixed incorrect Map value reading when key/value is of DATE or DATETIME type [doris-flink-connector]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on code in PR #419:
URL: https://github.com/apache/doris-flink-connector/pull/419#discussion_r1671984142
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/serialization/RowBatch.java:
##########
@@ -478,23 +486,38 @@ public boolean doConvert(
return true;
}
+ private Object handleMapFieldReader(FieldReader reader) {
+ if (reader instanceof TimeStampMicroReaderImpl) {
+ return longToLocalDateTime(reader.readLong());
+ }
+ if (reader instanceof DateDayReaderImpl) {
+ return LocalDate.ofEpochDay(((Integer) reader.readObject()).longValue());
+ }
+ return reader.readObject();
+ }
+
@VisibleForTesting
public LocalDateTime getDateTime(int rowIndex, FieldVector fieldVector) {
TimeStampMicroVector vector = (TimeStampMicroVector) fieldVector;
if (vector.isNull(rowIndex)) {
return null;
}
long time = vector.get(rowIndex);
+ return longToLocalDateTime(time);
+ }
+
+ @VisibleForTesting
+ public static LocalDateTime longToLocalDateTime(long time) {
Review Comment:
Is it possible to add a unit test for this method?
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/serialization/RowBatch.java:
##########
@@ -478,23 +486,38 @@ public boolean doConvert(
return true;
}
+ private Object handleMapFieldReader(FieldReader reader) {
+ if (reader instanceof TimeStampMicroReaderImpl) {
+ return longToLocalDateTime(reader.readLong());
+ }
+ if (reader instanceof DateDayReaderImpl) {
+ return LocalDate.ofEpochDay(((Integer) reader.readObject()).longValue());
+ }
+ return reader.readObject();
+ }
+
@VisibleForTesting
public LocalDateTime getDateTime(int rowIndex, FieldVector fieldVector) {
TimeStampMicroVector vector = (TimeStampMicroVector) fieldVector;
if (vector.isNull(rowIndex)) {
return null;
}
long time = vector.get(rowIndex);
+ return longToLocalDateTime(time);
+ }
+
+ @VisibleForTesting
+ public static LocalDateTime longToLocalDateTime(long time) {
Instant instant;
- if (time / 10000000000L == 0) { // datetime(0)
+ // 判断时间戳精度并处理
Review Comment:
Please use English comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](source) Fixed incorrect Map value reading when key/value is of DATE or DATETIME type [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #419:
URL: https://github.com/apache/doris-flink-connector/pull/419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org