[jira] [Resolved] (SPARK-30739) unable to turn off Hadoop's trash feature

["Ohad Raviv (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-30739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ohad Raviv resolved SPARK-30739.
--------------------------------
    Resolution: Workaround

> unable to turn off Hadoop's trash feature
> -----------------------------------------
>
>                 Key: SPARK-30739
>                 URL: https://issues.apache.org/jira/browse/SPARK-30739
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Ohad Raviv
>            Priority: Minor
>
> We're trying to turn off the `TrashPolicyDefault` in one of our Spark applications by setting `spark.hadoop.fs.trash.interval=0`, but it just stays `360` as configured in our cluster's `core-site.xml`.
> Trying to debug it we managed to set `spark.hadoop.fs.trash.classname=OtherTrashPolicy` and it worked. the main difference seems to be that `spark.hadoop.fs.trash.classname` does not appear in any of the `*-site.xml` files.
> when we print the conf that get initialized in `TrashPolicyDefault` we get:
> ```
> Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml, org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@561f0431, file:/hadoop03/yarn/local/usercache/.../hive-site.xml
> ```
> and:
> `fs.trash.interval=360 [programatically]`
> `fs.trash.classname=OtherTrashPolicy [programatically]`
>  
> any idea why `fs.trash.classname` works but `fs.trash.interval` doesn't?
> this seems maybe related to: -SPARK-9825.-
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org