[jira] [Created] (SPARK-14741) Streaming from partitioned directory structure captures unintended partition columns

["Tathagata Das (JIRA)" <ji...@apache.org>]

Tathagata Das created SPARK-14741:
-------------------------------------

             Summary: Streaming from partitioned directory structure captures unintended partition columns
                 Key: SPARK-14741
                 URL: https://issues.apache.org/jira/browse/SPARK-14741
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Tathagata Das
            Assignee: Tathagata Das


Consider the following directory structure

    dir/col=X/some-files

If we create a text format streaming dataframe on {dir/col=X/}, then it should not consider as partitioning in columns. Even though the streaming dataframe does not do so, the generated batch dataframes pick up col as a partitioning columns, causing mismatch streaming source schema and generated df schema. This leads to runtime failure: 


18:55:11.262 ERROR org.apache.spark.sql.execution.streaming.StreamExecution: Query query-0 terminated with error
java.lang.AssertionError: assertion failed: Invalid batch: c#2 != c#7,type#8





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org