You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2017/07/20 18:22:00 UTC

[jira] [Resolved] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

     [ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shixiong Zhu resolved SPARK-21463.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0

> Output of StructuredStreaming tables don't respect user specified schema when reading back the table
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21463
>                 URL: https://issues.apache.org/jira/browse/SPARK-21463
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL, Structured Streaming
>    Affects Versions: 2.2.0
>            Reporter: Burak Yavuz
>            Assignee: Burak Yavuz
>             Fix For: 2.3.0
>
>
> When using the MetadataLogFileIndex to read back a table, we don't respect the user provided schema as the proper column types. This can lead to issues when trying to read strings that look like dates that get truncated to DateType, or longs being truncated to IntegerType, just because a long value doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org