[jira] [Commented] (FLINK-19685) When use HBase-Connector lookupFunction, 8 hours less to query `DATE`,`TIME`,`TIMESTAMP` data

["Jark Wu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216141#comment-17216141 ] 

Jark Wu commented on FLINK-19685:
---------------------------------

Hi [~caozhen1937], HBase is differnet from JDBC, because HBase doesn't define TIMESTAMP type, it just stores bytes. The conversion between bytes and TIMESTAMP is defined by external systems, Flink define it in the docs[1].

As the docs describe, we will stores the milliseconds since epoch (you can think of it as UTC timestamp) for TIMESTAMP type . So if you are using Flink for reading and writing, all the things should be fine. However, if you are writing HBase by yourself, be careful to serialize TIMESTAMP values. 

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/hbase.html#data-type-mapping

> When use HBase-Connector lookupFunction, 8 hours less to query `DATE`,`TIME`,`TIMESTAMP` data 
> ----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-19685
>                 URL: https://issues.apache.org/jira/browse/FLINK-19685
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / HBase
>            Reporter: CaoZhen
>            Priority: Major
>
> from the code:
> HBaseSerde#createFieldDecoder
> {code:java}
> case DATE:
> case INTERVAL_YEAR_MONTH:
>    return Bytes::toInt;
> case TIME_WITHOUT_TIME_ZONE:
>    ......
>    return Bytes::toInt;
> case TIMESTAMP_WITHOUT_TIME_ZONE:
> case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
>    ......
>    return createTimestampDecoder();
> private static FieldDecoder createTimestampDecoder() {
>    return value -> {
>       // TODO: support higher precision
>       long milliseconds = Bytes.toLong(value);
>       return TimestampData.fromEpochMillis(milliseconds);
>    };
> }
> {code}
>  
> I think we should learn from JDBC's approach to dealing with time types.
> {code:java}
> case DATE:
>     return val -> (int) (((Date) val).toLocalDate().toEpochDay());
> case TIME_WITHOUT_TIME_ZONE:
>     return val -> (int) (((Time) val).toLocalTime().toNanoOfDay() / 1_000_000L);
> case TIMESTAMP_WITH_TIME_ZONE:
> case TIMESTAMP_WITHOUT_TIME_ZONE:
>     return val -> TimestampData.fromTimestamp((Timestamp) val);
> {code}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)