You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vadim Bichutskiy <va...@gmail.com> on 2015/05/14 22:55:38 UTC

textFileStream Question

How does textFileStream work behind the scenes? How does Spark Streaming
know what files are new and need to be processed? Is it based on time
stamp, file name?

Thanks,
Vadim
ᐧ

Re: textFileStream Question

Posted by Vadim Bichutskiy <va...@gmail.com>.
This is cool. Thanks Akhil.

ᐧ

On Sun, May 17, 2015 at 11:25 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> With file timestamp, you can actually see the finding new files logic
> from here
> <https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172>
>
> Thanks
> Best Regards
>
> On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy <
> vadim.bichutskiy@gmail.com> wrote:
>
>> How does textFileStream work behind the scenes? How does Spark Streaming
>> know what files are new and need to be processed? Is it based on time
>> stamp, file name?
>>
>> Thanks,
>> Vadim
>> ᐧ
>>
>
>

Re: textFileStream Question

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
With file timestamp, you can actually see the finding new files logic from
here
<https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172>

Thanks
Best Regards

On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy <
vadim.bichutskiy@gmail.com> wrote:

> How does textFileStream work behind the scenes? How does Spark Streaming
> know what files are new and need to be processed? Is it based on time
> stamp, file name?
>
> Thanks,
> Vadim
> ᐧ
>

回复：textFileStream Question

Posted by 董帅阳 <91...@qq.com>.
file timestamp




------------------ 原始邮件 ------------------
发件人: "Vadim Bichutskiy";<va...@gmail.com>;
发送时间: 2015年5月15日(星期五) 凌晨4:55
收件人: "user@spark.apache.org"<us...@spark.apache.org>; 

主题: textFileStream Question



How does textFileStream work behind the scenes? How does Spark Streaming know what files are new and need to be processed? Is it based on time stamp, file name?

Thanks,
Vadim
ᐧ