You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by 萝卜丝炒饭 <14...@qq.com> on 2017/02/23 09:07:33 UTC
Filestream can not recognize the copied file
Hi all,
I tested filestream today, my code looks like:
val fs = ssc.textFileStream(*)
erroelines = fs.filter( _.contains("erroe"))
erroelines.print
ssc.start()
when I edit a file and save it to the dir, it works well.
If i copy a file to the dir, it does work.
my issues are:
1, is it OK please? I means whether the code was designed to work as this.
2,which class monitor the dir pls?
Re: Filestream can not recognize the copied file
Posted by 萝卜丝炒饭 <14...@qq.com>.
hi all,
I checked the code just now.
Spark scan the dir and filter files by timestamp of file.
I copied files to the dir but not changed the timestamp.
I know the reason now.
thanks
---Original---
From: "萝卜丝炒饭"<14...@qq.com>
Date: 2017/2/23 17:07:33
To: "dev"<de...@spark.apache.org>;
Subject: Filestream can not recognize the copied file
Hi all,
I tested filestream today, my code looks like:
val fs = ssc.textFileStream(*)
erroelines = fs.filter( _.contains("erroe"))
erroelines.print
ssc.start()
when I edit a file and save it to the dir, it works well.
If i copy a file to the dir, it does work.
my issues are:
1, is it OK please? I means whether the code was designed to work as this.
2,which class monitor the dir pls?