[jira] [Commented] (ORC-526) orc-tools convert does not respect second fractions

["Yukihiro Okada (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920183#comment-16920183 ] 

Yukihiro Okada commented on ORC-526:
------------------------------------

 Hi [~fgroffenorcl] ,

I send a PR. Is this PR the same as your intention?

Could you please take a look?
{code:java}
% cat t/some.json 
{"myid":1,"time":"2019-10-28 07:34:07.123456"} 
{"myid":2,"time":"2019-10-29 07:20:57.567891"}

% if [ -f output.orc ]; then rm -f output.orc ; fi && java -jar tools/target/orc-tools-1.6.0-SNAPSHOT-uber.jar convert --schema "struct<id:int,time:timestamp>"  --timestampformat "yyyy-MM-dd HH:mm:ss.SSSSSS" t/some.json && java -jar tools/target/orc-tools-1.6.0-SNAPSHOT-uber.jar data output.orc
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing t/some.json
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing data file output.orc [length: 327]
{"id":null,"time":"2019-10-28 07:34:07.123456"}
{"id":null,"time":"2019-10-29 07:20:57.567891"}
________________________________________________________________________________________________________________________

{code}

> orc-tools convert does not respect second fractions
> ---------------------------------------------------
>
>                 Key: ORC-526
>                 URL: https://issues.apache.org/jira/browse/ORC-526
>             Project: ORC
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 1.2.0, 1.5.5, 1.6.0
>            Reporter: Fabian Groffen
>            Priority: Trivial
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{% cat timestamp2.csv }}
> {{2019-01-01 00:00:00.0000}}
> {{2015-01-01 00:00:00.0001}}
> {{2015-01-01 00:00:00.0000}}
> {{2014-12-31 23:59:59.9999}}
> {{1970-01-01 00:00:00.0001}}
> {{1970-01-01 00:00:00.0000}}
> {{1969-12-31 23:59:59.9999}}
> {{1969-12-31 23:59:59.0001}}
> {{1969-12-31 23:59:59.0000}}
> {{1969-12-31 23:59:58.9999}}
> {{% java -jar tools/target/orc-tools-1.6.0-SNAPSHOT-uber.jar convert --schema timestamp --timestampformat "yyyy-MM-dd HH:mm:ss.SSSS" timestamp2.csv -o timestamp2-tools16.orc}}
> Reading back the file generated above excludes the second fractions (nanos).
>  
> Reason for this is that tools/convert/CsvReader.java (as well as tools/convert/JsonReader.java) do not consider the nanos.  They just create a Timestamp and never call setNanos() method on it (in this case with the result of a getNano() call to the ZonedDateTime.  This needs to be done in 4 places.  Unfortunately I am not allowed to make a contribution at this point, so I can only describe the problem.  I can confirm with a change/fix like that, the nanos are stored in the resulting ORC file.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)