RE: Unable to correctly change umask for parquet files with Spark and Zeppelin

[Anton Bubna-Litic <An...@quantium.com.au>]

Hey,

Has anybody experienced this problem?

________________________________


Hi guys,

I would like to be able to set the umask settings for when I am writing a parquet file in spark through Zeppelin. I can set the spark settings:

*         spark.sparkContext.hadoopConfiguration.set("fs.permissions.umask-mode", "007")

Unfortunately, when it writes the parent folder and the _success files use the OS file system which I can't change from Zeppelin. If I am running spark shell, I can set it in my .profile folder and it will work. However, we run dockerised Zeppelin and it will always revert to the global umask.

Is there a way to fix this?

Cheers,
Anton


Anton Bubna-Litic
QUANTIUM
Level 25, 8-12 Chifley Square
Sydney NSW 2000
Australia

T: +61 2 8222 3585

W: quantium.com.au<http://www.quantium.com>
________________________________
linkedin.com/company/quantium<http://www.linkedin.com/company/quantium>
facebook.com/QuantiumAustralia<http://www.facebook.com/QuantiumAustralia>
twitter.com/QuantiumAU<http://www.twitter.com/QuantiumAU>

The contents of this email, including attachments, may be confidential information. If you are not the intended recipient, any use, disclosure or copying of the information is unauthorised. If you have received this email in error, we would be grateful if you would notify us immediately by email reply, phone (+ 61 2 9292 6400) or fax (+ 61 2 9292 6444) and delete the message from your system.