You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "seth (JIRA)" <ji...@apache.org> on 2016/04/18 13:30:25 UTC
[jira] [Comment Edited] (SPARK-14688) pyspark textFileStream
gzipped
[ https://issues.apache.org/jira/browse/SPARK-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245485#comment-15245485 ]
seth edited comment on SPARK-14688 at 4/18/16 11:29 AM:
--------------------------------------------------------
A correction :
The Scala method also doesn't handle gzip.
You can check this with:
#python
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
ssc = StreamingContext(sc, 10)
stream = ssc..textFileStream('dir/')
winStream.pprint()
ssc.start()
ssc.awaitTermination()
#scala
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
val ssc = new StreamingContext(sc, Seconds(5))
val stream = ssc.textFileStream("dir/")
stream.print()
ssc.start()
ssc.awaitTermination()
was (Author: seth_123321):
A correction :
The Scala method also doesn't handle gzip.
You can check this with:
{{
#python
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
ssc = StreamingContext(sc, 10)
stream = ssc..textFileStream('dir/')
winStream.pprint()
ssc.start()
ssc.awaitTermination()
#scala
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
val ssc = new StreamingContext(sc, Seconds(5))
val stream = ssc.textFileStream("dir/")
stream.print()
ssc.start()
ssc.awaitTermination()
}}
> pyspark textFileStream gzipped
> ------------------------------
>
> Key: SPARK-14688
> URL: https://issues.apache.org/jira/browse/SPARK-14688
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Streaming
> Affects Versions: 1.6.1
> Reporter: seth
> Labels: pyspark, streaming
>
> pyspark streamingObject does not support reading gzip files.
> 2 notes:
> 1.regular sparkContext does support gzip files
> 2. Java/Scala method support streaming gzip files
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org