[jira] [Resolved] (SPARK-45855) Unable to set compression codec for Hive CTAS

["Tim Robertson (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Robertson resolved SPARK-45855.
-----------------------------------
    Resolution: Fixed

I found this is fixed in 3.5.0 and I strongly suspect is caused by the same thing documented and fixed in  #43504 

> Unable to set compression codec for Hive CTAS
> ---------------------------------------------
>
>                 Key: SPARK-45855
>                 URL: https://issues.apache.org/jira/browse/SPARK-45855
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.4.0
>         Environment: Spark 3.4.0 
> Stackable.tech release 23.7.0 which runs spark on K8s.
>            Reporter: Tim Robertson
>            Priority: Major
>             Fix For: 3.5.0
>
>
> Hi,
> We've discovered code that worked in Spark 3.3.0 doesn't in 3.4.0. I can't find anything in the release notes to indicate why, so I wonder if this is a bug. Thank you for looking.
> Here we're using our own custom codec, but we noticed we can't set gzip either.
> {{  SparkConf conf = spark.sparkContext().conf();}}
> {{  conf.set("hive.exec.compress.output", "true");}}
> {{  conf.set("mapred.output.compression.codec", D2Codec.class.getName()); }}
> {{  spark.sql("CREATE TABLE b AS SELECT id FROM a");}}
> This will create the table, but it writes uncompressed files, where Spark 3.3.0 would write compressed files. 
> Any advice is appreciated and I can help run tests. We run Spark on K8S using the stackable.tech distribution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org