You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by LinQili <li...@outlook.com> on 2015/01/22 07:45:44 UTC

Is there a way to delete hdfs file/directory using spark API?

Hi, allI wonder how to delete hdfs file/directory using spark API? 		 	   		  

Re: Is there a way to delete hdfs file/directory using spark API?

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
There is no direct way of doing it, but you can do something like this:

val hadoopConf = ssc.sparkContext.hadoopConfiguration

var hdfs = org.apache.hadoop.fs.FileSystem.get(hadoopConf)


tmp_stream = ssc.textFileStream("/akhld/sigmoid/") // each line will have
hdfs location to be deleted.


tmp_stream.foreachRDD(path => {


    try {

      hdfs.delete(new org.apache.hadoop.fs.Path(path), true)

    } catch{ case e: Exception =>

      println("w00t!! Exception!!HDFS => " + e)

    }



  })


Thanks
Best Regards

On Thu, Jan 22, 2015 at 12:15 PM, LinQili <li...@outlook.com> wrote:

> Hi, all
> I wonder how to delete hdfs file/directory using spark API?
>

Re: Is there a way to delete hdfs file/directory using spark API?

Posted by Raghavendra Pandey <ra...@gmail.com>.
You can use Hadoop Client Api to remove files
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#delete(org.apache.hadoop.fs.Path,
boolean). I don't think spark has any wrapper on hadoop filesystem APIs.

On Thu, Jan 22, 2015 at 12:15 PM, LinQili <li...@outlook.com> wrote:

> Hi, all
> I wonder how to delete hdfs file/directory using spark API?
>