[jira] [Commented] (ORC-236) orc-tools JsonReader does not support uniontype in schema

["Riza Suminto (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480784#comment-17480784 ] 

Riza Suminto commented on ORC-236:
----------------------------------

Hello! I'm currently hacking convert tool to support UNION.

Given the Union type specification at [https://cwiki.apache.org/confluence/display/hive/languagemanual+types#LanguageManualTypes-UnionTypesunionUnionTypes] , I think we should include the tag information in the json source file.

This is what has work in my hack:

 
{code:java}
$ cat sample.json
{"foo": {0 : 1}}
{"foo": {1 : "testing"}}
{"foo": {0 : 3}}
$ java -jar ./java/tools/target/orc-tools-1.8.0-SNAPSHOT-uber.jar convert -o sample.orc -s "struct<foo:uniontype<int,string>>" sample.json
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing sample.json
$ ./build/tools/src/orc-contents sample.orc
{"foo": {"tag": 0, "value": 1}}
{"foo": {"tag": 1, "value": "testing"}}
{"foo": {"tag": 0, "value": 3}}
{code}
 

 

> orc-tools JsonReader does not support uniontype in schema
> ---------------------------------------------------------
>
>                 Key: ORC-236
>                 URL: https://issues.apache.org/jira/browse/ORC-236
>             Project: ORC
>          Issue Type: Bug
>            Reporter: Anthony Hsu
>            Priority: Major
>
> To repro:
> {noformat}
> $ cat sample.json
> {"foo":1}
> $ java -jar ~/lib/orc-tools-1.4.0-uber.jar convert -s "struct<foo:uniontype<int,string>>" -o output.orc sample.json
> log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Processing sample.json
> Exception in thread "main" java.lang.IllegalArgumentException: Unhandled type uniontype<int,string>
> 	at org.apache.orc.tools.convert.JsonReader.createConverter(JsonReader.java:233)
> 	at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:260)
> 	at org.apache.orc.tools.convert.ConvertTool.main(ConvertTool.java:67)
> 	at org.apache.orc.tools.Driver.main(Driver.java:112)
> {noformat}
> Seems like [JsonReader.createConverter|https://github.com/apache/orc/blob/635833d13ff4a5c6a4ea3d73bc41af3efaf816cd/java/tools/src/java/org/apache/orc/tools/convert/JsonReader.java#L224] does not support [UNION|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/TypeDescription.java#L116].



--
This message was sent by Atlassian Jira
(v8.20.1#820001)