[jira] [Assigned] (SPARK-42439) Job description in v2 FileWrites can have the wrong committer

["Apache Spark (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-42439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-42439:
------------------------------------

    Assignee: Apache Spark

> Job description in v2 FileWrites can have the wrong committer
> -------------------------------------------------------------
>
>                 Key: SPARK-42439
>                 URL: https://issues.apache.org/jira/browse/SPARK-42439
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.3.1
>            Reporter: Lorenzo Martini
>            Assignee: Apache Spark
>            Priority: Minor
>
> There is a difference in behavior between v1 writes and v2 writes in the order of events happening when configuring the file writer and the committer.
> v1:
>  # writer.prepareWrite()
>  # committer.setupJob()
> v2:
>  # committer.setupJob()
>  # writer.prepareWrite()
>  
> This is because the `prepareWrite()` call (that is the one performing the call `
> job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])`)
> happens as part of the `createWriteJobDescription` which is `lazy val` in the `toBatch` call and therefore is evaluated after the `committer.setupJob` at the end of the `toBatch`
> This causes issues when evaluating the committer as some elements might be missing, for example the aforementioned output format class not being set, causing the committer being set up as generic write instead of parquet write.
>  
> The fix is very simple and it is to make the `createJobDescription` call non-lazy



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org