You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/09/08 06:33:28 UTC
FileStreamSource source checks path eagerly?
Hi,
I'm wondering what's the rationale for checking the path option
eagerly in FileStreamSource? My thinking is that until start is called
there's no processing going on that is supposed to happen on executors
(not the driver) with the path available.
I could (and perhaps should) use dfs but IMHO that just hides the real
question of the text source eagerness.
Please help me understand the rationale of the choice. Thanks!
scala> spark.version
res0: String = 2.1.0-SNAPSHOT
scala> spark.readStream.format("text").load("/var/logs")
org.apache.spark.sql.AnalysisException: Path does not exist: /var/logs;
at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:229)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:81)
at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:81)
at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
... 48 elided
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FileStreamSource source checks path eagerly?
Posted by Matei Zaharia <ma...@gmail.com>.
This source is meant to be used for a shared file system such as HDFS or NFS, where both the driver and the workers can see the same folders. There's no support in Spark for just working with local files on different workers.
Matei
> On Sep 8, 2016, at 2:23 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>
> Hi Steve,
>
> Thank you for more source-oriented answer. Helped but didn't explain
> the reason for such eagerness. The file(s) might not be on the driver
> but on executors only where the Spark job(s) run. I don't see why
> Spark should check the file(s) regardless of glob pattern being used.
>
> You see my way of thinking?
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Thu, Sep 8, 2016 at 11:20 AM, Steve Loughran <st...@hortonworks.com> wrote:
>> failfast generally means that you find problems sooner rather than later, and here, potentially, that your code runs but simply returns empty data without any obvious cue as to what is wrong.
>>
>> As is always good in OSS, follow those stack trace links to see what they say:
>>
>> // Check whether the path exists if it is not a glob pattern.
>> // For glob pattern, we do not check it because the glob pattern might only make sense
>> // once the streaming job starts and some upstream source starts dropping data.
>>
>> If you specify a glob pattern, you'll get the late check at the expense of the risk of that empty data source if the pattern is wrong. Something like "/var/log\s" would suffice, as the presence of the backslash is enough for SparkHadoopUtil.isGlobPath() to conclude that its something for the globber.
>>
>>
>>> On 8 Sep 2016, at 07:33, Jacek Laskowski <ja...@japila.pl> wrote:
>>>
>>> Hi,
>>>
>>> I'm wondering what's the rationale for checking the path option
>>> eagerly in FileStreamSource? My thinking is that until start is called
>>> there's no processing going on that is supposed to happen on executors
>>> (not the driver) with the path available.
>>>
>>> I could (and perhaps should) use dfs but IMHO that just hides the real
>>> question of the text source eagerness.
>>>
>>> Please help me understand the rationale of the choice. Thanks!
>>>
>>> scala> spark.version
>>> res0: String = 2.1.0-SNAPSHOT
>>>
>>> scala> spark.readStream.format("text").load("/var/logs")
>>> org.apache.spark.sql.AnalysisException: Path does not exist: /var/logs;
>>> at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:229)
>>> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:81)
>>> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:81)
>>> at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
>>> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
>>> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
>>> ... 48 elided
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FileStreamSource source checks path eagerly?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Steve,
Thank you for more source-oriented answer. Helped but didn't explain
the reason for such eagerness. The file(s) might not be on the driver
but on executors only where the Spark job(s) run. I don't see why
Spark should check the file(s) regardless of glob pattern being used.
You see my way of thinking?
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Thu, Sep 8, 2016 at 11:20 AM, Steve Loughran <st...@hortonworks.com> wrote:
> failfast generally means that you find problems sooner rather than later, and here, potentially, that your code runs but simply returns empty data without any obvious cue as to what is wrong.
>
> As is always good in OSS, follow those stack trace links to see what they say:
>
> // Check whether the path exists if it is not a glob pattern.
> // For glob pattern, we do not check it because the glob pattern might only make sense
> // once the streaming job starts and some upstream source starts dropping data.
>
> If you specify a glob pattern, you'll get the late check at the expense of the risk of that empty data source if the pattern is wrong. Something like "/var/log\s" would suffice, as the presence of the backslash is enough for SparkHadoopUtil.isGlobPath() to conclude that its something for the globber.
>
>
>> On 8 Sep 2016, at 07:33, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi,
>>
>> I'm wondering what's the rationale for checking the path option
>> eagerly in FileStreamSource? My thinking is that until start is called
>> there's no processing going on that is supposed to happen on executors
>> (not the driver) with the path available.
>>
>> I could (and perhaps should) use dfs but IMHO that just hides the real
>> question of the text source eagerness.
>>
>> Please help me understand the rationale of the choice. Thanks!
>>
>> scala> spark.version
>> res0: String = 2.1.0-SNAPSHOT
>>
>> scala> spark.readStream.format("text").load("/var/logs")
>> org.apache.spark.sql.AnalysisException: Path does not exist: /var/logs;
>> at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:229)
>> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:81)
>> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:81)
>> at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
>> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
>> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
>> ... 48 elided
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FileStreamSource source checks path eagerly?
Posted by Steve Loughran <st...@hortonworks.com>.
failfast generally means that you find problems sooner rather than later, and here, potentially, that your code runs but simply returns empty data without any obvious cue as to what is wrong.
As is always good in OSS, follow those stack trace links to see what they say:
// Check whether the path exists if it is not a glob pattern.
// For glob pattern, we do not check it because the glob pattern might only make sense
// once the streaming job starts and some upstream source starts dropping data.
If you specify a glob pattern, you'll get the late check at the expense of the risk of that empty data source if the pattern is wrong. Something like "/var/log\s" would suffice, as the presence of the backslash is enough for SparkHadoopUtil.isGlobPath() to conclude that its something for the globber.
> On 8 Sep 2016, at 07:33, Jacek Laskowski <ja...@japila.pl> wrote:
>
> Hi,
>
> I'm wondering what's the rationale for checking the path option
> eagerly in FileStreamSource? My thinking is that until start is called
> there's no processing going on that is supposed to happen on executors
> (not the driver) with the path available.
>
> I could (and perhaps should) use dfs but IMHO that just hides the real
> question of the text source eagerness.
>
> Please help me understand the rationale of the choice. Thanks!
>
> scala> spark.version
> res0: String = 2.1.0-SNAPSHOT
>
> scala> spark.readStream.format("text").load("/var/logs")
> org.apache.spark.sql.AnalysisException: Path does not exist: /var/logs;
> at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:229)
> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:81)
> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:81)
> at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
> at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
> ... 48 elided
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FileStreamSource source checks path eagerly?
Posted by Jacek Laskowski <ja...@japila.pl>.
On Thu, Sep 8, 2016 at 9:03 AM, Fred Reiss <fr...@gmail.com> wrote:
> I suppose the type-inference-time check for the presence of the input
> directory could be moved to the FileStreamSource's initialization. But if
> the directory isn't there when the source is being created, it probably
> won't be there when the source is instantiated.
Hi Fred,
Thanks for your prompt response, Fred.
Isn't it opposite to sc.textFile? The source might not be available
until load. There's no reason it should. Yet it is definitely not
against the "contract" of DataFrameReader.textFile and perhaps it's
implictly assumed in SQL.
scala> spark.read.textFile("whatever")
org.apache.spark.sql.AnalysisException: Path does not exist:
file:/Users/jacek/dev/oss/spark/whatever;
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:371)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:360)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:344)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:360)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:500)
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:536)
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:509)
... 48 elided
I thought it might've been due to schema inference but...
scala> spark.read.schema(StructType(Seq())).textFile("whatever")
org.apache.spark.sql.AnalysisException: User specified schema not
supported with `textFile`;
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:534)
at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:509)
... 50 elided
(which also confuses me, but don't wanna drag this thread in multiple
directions) Definitely need some help to understand the rationale
behing this eager behaviour.
Thanks!
Pozdrawiam,
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FileStreamSource source checks path eagerly?
Posted by Fred Reiss <fr...@gmail.com>.
The input directory does need to be visible from the driver process, since
FileStreamSource does its polling from the driver. FileStreamSource creates
a Dataset for each microbatch.
I suppose the type-inference-time check for the presence of the input
directory could be moved to the FileStreamSource's initialization. But if
the directory isn't there when the source is being created, it probably
won't be there when the source is instantiated.
Fred
On Wed, Sep 7, 2016 at 11:33 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> I'm wondering what's the rationale for checking the path option
> eagerly in FileStreamSource? My thinking is that until start is called
> there's no processing going on that is supposed to happen on executors
> (not the driver) with the path available.
>
> I could (and perhaps should) use dfs but IMHO that just hides the real
> question of the text source eagerness.
>
> Please help me understand the rationale of the choice. Thanks!
>
> scala> spark.version
> res0: String = 2.1.0-SNAPSHOT
>
> scala> spark.readStream.format("text").load("/var/logs")
> org.apache.spark.sql.AnalysisException: Path does not exist: /var/logs;
> at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(
> DataSource.scala:229)
> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$
> lzycompute(DataSource.scala:81)
> at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(
> DataSource.scala:81)
> at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(
> StreamingRelation.scala:30)
> at org.apache.spark.sql.streaming.DataStreamReader.
> load(DataStreamReader.scala:142)
> at org.apache.spark.sql.streaming.DataStreamReader.
> load(DataStreamReader.scala:153)
> ... 48 elided
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>