You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by LinQili <li...@outlook.com> on 2015/01/22 07:45:44 UTC
Is there a way to delete hdfs file/directory using spark API?
Hi, allI wonder how to delete hdfs file/directory using spark API?
Re: Is there a way to delete hdfs file/directory using spark API?
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
There is no direct way of doing it, but you can do something like this:
val hadoopConf = ssc.sparkContext.hadoopConfiguration
var hdfs = org.apache.hadoop.fs.FileSystem.get(hadoopConf)
tmp_stream = ssc.textFileStream("/akhld/sigmoid/") // each line will have
hdfs location to be deleted.
tmp_stream.foreachRDD(path => {
try {
hdfs.delete(new org.apache.hadoop.fs.Path(path), true)
} catch{ case e: Exception =>
println("w00t!! Exception!!HDFS => " + e)
}
})
Thanks
Best Regards
On Thu, Jan 22, 2015 at 12:15 PM, LinQili <li...@outlook.com> wrote:
> Hi, all
> I wonder how to delete hdfs file/directory using spark API?
>
Re: Is there a way to delete hdfs file/directory using spark API?
Posted by Raghavendra Pandey <ra...@gmail.com>.
You can use Hadoop Client Api to remove files
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#delete(org.apache.hadoop.fs.Path,
boolean). I don't think spark has any wrapper on hadoop filesystem APIs.
On Thu, Jan 22, 2015 at 12:15 PM, LinQili <li...@outlook.com> wrote:
> Hi, all
> I wonder how to delete hdfs file/directory using spark API?
>