[jira] [Commented] (SPARK-14688) pyspark textFileStream gzipped

["seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266633#comment-15266633 ] 

seth commented on SPARK-14688:
------------------------------

ok, so I notices the following behavior: the successful outcome is dependent on the compression type.
so everything is fine if you created the file with {{gzip file.txt}}
but it is not ok if one uses {{tar -czvf file.txt}}.

Can you please verify this 

> pyspark textFileStream gzipped
> ------------------------------
>
>                 Key: SPARK-14688
>                 URL: https://issues.apache.org/jira/browse/SPARK-14688
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Streaming
>    Affects Versions: 1.6.1
>            Reporter: seth
>              Labels: pyspark, streaming
>
> pyspark streamingObject does not support reading gzip files. 
> Note that regular sparkContext does support gzip files
> [edited]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org