[jira] [Commented] (SPARK-21019) read orc when some of the columns are missing in some files

["Mahesh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046485#comment-16046485 ] 

Mahesh commented on SPARK-21019:
--------------------------------

Can you include the exception in your defect, from what I know spark uses orc implementation that is hosted at http://orc.apache.org/
You may want to try a similar thing with that library. I will try to run your code.

>  read orc when some of the columns are missing in some files
> ------------------------------------------------------------
>
>                 Key: SPARK-21019
>                 URL: https://issues.apache.org/jira/browse/SPARK-21019
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>    Affects Versions: 2.1.1
>            Reporter: Modi Tamam
>
> I'm using Spark-2.1.1.
> I'm experiencing an issue when I'm reading a bunch of ORC files when some of the fields are missing from some of the files (file-1 has fields 'a' and 'b', file-2 has fields 'a' and 'c'). When I'm running the same flow with JSON files format, every thing is just fine (you can see it at the code snippets , if you'll run it...) My question is whether it's a bug or an expected behavior?
> I'v pushed a maven project, ready for run, you can find it here https://github.com/MordechaiTamam/spark-orc-issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org