[jira] [Commented] (HIVE-24693) Parquet Timestamp Values Read/Write Very Slow

["David Mollitor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-24693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275222#comment-17275222 ] 

David Mollitor commented on HIVE-24693:
---------------------------------------

Hitting a weird issue.  There's a unit test that goes from Date to Timestamp that is failing.  It seems that the Timestamp value and the toString do not agree with each other:

 
{code:java}
  public static void main(String[] args) throws IOException, URISyntaxException
  {
    DateTimeFormatterBuilder builder = new DateTimeFormatterBuilder();
    // Date and time parts
    builder.append(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"));
    // Fractional part
    builder.optionalStart().appendFraction(ChronoField.NANO_OF_SECOND, 0, 9, true).optionalEnd();
    DateTimeFormatter PRINT_FORMATTER = builder.toFormatter();

    int daysSinceEpoch = -1133938638;
    
    LocalDate localDate = LocalDate.ofEpochDay(daysSinceEpoch);
    long epochMillis = localDate.atStartOfDay().toInstant(ZoneOffset.UTC).toEpochMilli();
    
    LocalDateTime localDateTime = LocalDateTime.ofInstant(Instant.ofEpochMilli(epochMillis), ZoneOffset.UTC);
    
    System.out.println(epochMillis);
    System.out.println(localDate);
    System.out.println(localDateTime.format(PRINT_FORMATTER));
  }
{code}

It's a big negative number.  It should be a negative date, however,:

{code:none}
-97972298323200000
-3102649-06-17
+3102650-06-17 00:00:00
{code}

For some reason, the print formatted date is different that the toString value.  Hmmm.

> Parquet Timestamp Values Read/Write Very Slow
> ---------------------------------------------
>
>                 Key: HIVE-24693
>                 URL: https://issues.apache.org/jira/browse/HIVE-24693
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: David Mollitor
>            Assignee: David Mollitor
>            Priority: Critical
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Parquet {{DataWriteableWriter}} relias on {{NanoTimeUtils}} to convert a timestamp object into a binary value.  The way in which it does this,... it calls {{toString()}} on the timestamp object, and then parses the String.  This particular timestamp do not carry a timezone, so the string is something like:
> {{2021-21-03 12:32:23.0000...}}
> The parse code tries to parse the string assuming there is a time zone, and if not, falls-back and applies the provided "default time zone".  As was noted in [HIVE-24353], if something fails to parse, it is very expensive to try to parse again.  So, for each timestamp in the Parquet file, it:
> * Builds a string from the time stamp
> * Parses it (throws an exception, parses again)
> There is no need to do this kind of string manipulations/parsing, it should just be using the epoch millis/seconds/time stored internal to the Timestamp object.
> {code:java}
>   // Converts Timestamp to TimestampTZ.
>   public static TimestampTZ convert(Timestamp ts, ZoneId defaultTimeZone) {
>     return parse(ts.toString(), defaultTimeZone);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)