You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sanket Reddy (Jira)" <ji...@apache.org> on 2020/01/02 16:27:00 UTC
[jira] [Created] (SPARK-30411) saveAsTable does not honor
spark.hadoop.hive.warehouse.subdir.inherit.perms
Sanket Reddy created SPARK-30411:
------------------------------------
Summary: saveAsTable does not honor spark.hadoop.hive.warehouse.subdir.inherit.perms
Key: SPARK-30411
URL: https://issues.apache.org/jira/browse/SPARK-30411
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.4
Reporter: Sanket Reddy
-bash-4.2$ hdfs dfs -ls /tmp | grep my_databases
drwxr-x--T - redsanket users 0 2019-12-04 20:15 /tmp/my_databases
>>>spark.sql("CREATE TABLE redsanket_db.example(bcookie string, ip int) STORED AS orc");
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
drwxr-x--T - redsanket users 0 2019-12-04 20:20 /tmp/my_databases/example
Now after saveAsTable
>>> data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)]
>>> df = spark.createDataFrame(data)
>>> df.write.format("orc").mode('overwrite').saveAsTable('redsanket_db.example')
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
drwx------ - redsanket users 0 2019-12-04 20:23 /tmp/my_databases/example
Overwrites the permissions
Insert into honors preserving parent directory permissions.
>>> spark.sql("DROP table redsanket_db.example");
DataFrame[]
>>> spark.sql("CREATE TABLE redsanket_db.example(bcookie string, ip int) STORED AS orc");
DataFrame[]
>>> df.write.format("orc").insertInto('redsanket_db.example')
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
drwxr-x--T - schintap users 0 2019-12-04 20:43 /tmp/my_databases/example
It is either limitation of the API based on the mode and the behavior has to be documented or needs to be fixed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org