You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "feiwang (Jira)" <ji...@apache.org> on 2019/09/15 17:13:00 UTC
[jira] [Comment Edited] (SPARK-28945) Allow concurrent writes to
different partitions with dynamic partition overwrite
[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930010#comment-16930010 ]
feiwang edited comment on SPARK-28945 at 9/15/19 5:12 PM:
----------------------------------------------------------
[~cloud_fan] [~advancedxy]
Hi, I think the exception shown in the email(https://mail-archives.apache.org/mod_mbox/spark-dev/201909.mbox/%3CCANx3uAinvf2LdtKfWUsykCJ%2BkHh6oYy0Pt_5LvcTSURGmQKQwg%40mail.gmail.com%3E) can be resolved by this PR: (https://github.com/apache/spark/pull/25795).
When dynamicPartitionOverwrite is true, we should skip commitJob.
was (Author: hzfeiwang):
[~cloud_fan] [~advancedxy]
Hi, I think the exception shown in the email https://mail-archives.apache.org/mod_mbox/spark-dev/201909.mbox/%3CCANx3uAinvf2LdtKfWUsykCJ%2BkHh6oYy0Pt_5LvcTSURGmQKQwg%40mail.gmail.com%3E can be resolved by this PR: https://github.com/apache/spark/pull/25795.
When dynamicPartitionOverwrite is true, we should skip commitJob.
> Allow concurrent writes to different partitions with dynamic partition overwrite
> --------------------------------------------------------------------------------
>
> Key: SPARK-28945
> URL: https://issues.apache.org/jira/browse/SPARK-28945
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: koert kuipers
> Priority: Minor
>
> It is desirable to run concurrent jobs that write to different partitions within same baseDir using partitionBy and dynamic partitionOverwriteMode.
> See for example here:
> https://stackoverflow.com/questions/38964736/multiple-spark-jobs-appending-parquet-data-to-same-base-path-with-partitioning
> Or the discussion here:
> https://github.com/delta-io/delta/issues/9
> This doesnt seem that difficult. I suspect only changes needed are in org.apache.spark.internal.io.HadoopMapReduceCommitProtocol, which already has a flag for dynamicPartitionOverwrite. I got a quick test to work by disabling all committer activity (committer.setupJob, committer.commitJob, etc.) when dynamicPartitionOverwrite is true.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org