[jira] [Issue Comment Deleted] (ORC-626) Reading Struct Column Having Multiple Fields With Same Name Causes java.io.EOFException

["Syed Shameerur Rahman (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ORC-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Syed Shameerur Rahman updated ORC-626:
--------------------------------------
    Comment: was deleted

(was: [~omalley] Gentle reminder!)

> Reading Struct Column Having Multiple Fields With Same Name Causes java.io.EOFException
> ---------------------------------------------------------------------------------------
>
>                 Key: ORC-626
>                 URL: https://issues.apache.org/jira/browse/ORC-626
>             Project: ORC
>          Issue Type: Bug
>            Reporter: Syed Shameerur Rahman
>            Priority: Major
>
> *Steps To Repro In Hive:*
> {code:java}
> set hive.fetch.task.conversion=none;
> set orc.force.positional.evolution=true;
> create table complex_orc(device struct<a:string,a:string,b:string>) stored as orc;
> insert into complex_orc select named_struct("a","123","a","823","b","23");
> select * from complex_orc;
> {code}
> *Fails with the following exception:*
> {code:java}
> Caused by: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 3 kind LENGTH position: 6 length: 6 range: 0 offset: 16 limit: 16 range 0 = 0 to 6 uncompressed: 3 to 3
> 	at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:61)
> 	at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:323)
> 	at org.apache.orc.impl.RunLengthIntegerReaderV2.nextVector(RunLengthIntegerReaderV2.java:369)
> 	at org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.commonReadByteArrays(TreeReaderFactory.java:1299)
> 	at org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.readOrcByteArrays(TreeReaderFactory.java:1336)
> 	at org.apache.orc.impl.TreeReaderFactory$StringDirectTreeReader.nextVector(TreeReaderFactory.java:1434)
> 	at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.nextVector(TreeReaderFactory.java:1280)
> 	at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1836)
> 	at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:1818)
> 	at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1149)
> {code}
> This is caused due to ORC-54 where schema evolution was done based on field names rather than index. Setting *orc.force.positional.evolution* will force to do a positional schema evolution but the positional level is hardcoded to 1 (for non acid). Even though it doesn't make sense to have multiple fields with same name in in struct, It breaks the backward compatibly with hive 1.2 / hive2.1.
> [~omalley] Can you please share the idea behind setting *positional level* to 1. Is it really required when orc.force.positional.evolution is set? I mean can't we just do positional schema evolution for all the levels when orc.force.positional.evolution is set.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)