You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "LvDongrong (JIRA)" <ji...@apache.org> on 2017/05/03 03:07:04 UTC

[jira] [Created] (SPARK-20572) Spark Streaming fail to read file on Hdfs

LvDongrong created SPARK-20572:
----------------------------------

             Summary: Spark Streaming fail to read file on Hdfs
                 Key: SPARK-20572
                 URL: https://issues.apache.org/jira/browse/SPARK-20572
             Project: Spark
          Issue Type: Bug
          Components: DStreams
    Affects Versions: 2.1.0
            Reporter: LvDongrong


When I move a file on hdfs to the target directory where Spark Streaming read from, Spark Streaming fail to read the file. Then I found the Spark Streaming Hdfs interface only read the file whose modtime is in the interval of the current batch,but when we move the file to the target directory,the modtime of the file is never changed. May be the method to find new file is not appropriate.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org